
A Resource Guide for Head Start Programs: 
Movi ng Beyond a Culture of Compliance to a 
Culture of Conti nuous I mprovement 


OPRE Report # 2015-02 
January 2015 



A Resource Guide for Head Start Programs: 
Moving Beyond a Culture of Compliance to a Culture 
of Continuous I mprovement 


J anuary 2015 


OPRE Report 2015-02 


Teresa Derrick-Mills, Mary K. Winkler, Olivia Healy, and Erica Greenberg 

Urban Institute 


Submitted to: 

Mary Bruce Webb and Jennifer Brooks, Project Officers 
Office of Planning, Research and Evaluation 
Administration for Children and Families 
U.S. Department of Health and Human Services 

Contract Number: HHSP23320095654WC - HHSP2337038T 

Project Director: Teresa Derrick- Mills 
The Urban I nstitute 
2100 M Street NW 
Washington, DC 20037 

This report is in the public domain. Permission to reproduce is not necessary. Suggested citation: 
Derrick-Mills, Teresa, Mary K. Winkler, Olivia Healy, and Erica Greenberg. (2015). A Resource Guide 
for Head Start Programs: Moving Beyond a Culture of Compliance to a Culture of Continuous 
Improvement, OPRE Report # 2015-02, Washington, DC: Office of Planning, Research and Evaluation, 
Administration for Children and Families, U.S. Department of Health and Human Services. 

Disclaimer 

The views expressed in this publication do not necessarily reflect the views or policies of the Office of 
Planning, Research and Evaluation, the Administration for Children and Families, or the U.S. 
Department of Health and Human Services. 

This report and other reports sponsored by the Office of Planning, Research and Evaluation are 
available at http://www.acf.hhs.gov/proqrams/opre/index.html . 


Cover photo ©iStock.com/CEFutcher. 







Acknowledgements 


We would like to acknowledge our project officer at the U.S. Department of Health 
and Human Services (DHHS), Office of Planning, Research and Evaluation (OPRE) 
Mary Bruce Webb, our former project officers Jennifer Brooks and Mary 
Mueggenborg, and Society for Research on Child Development Fellows Nina 
Philipsen Hetzner and Kelly Fisher. We also thank the Office of Head Start, 
particularly Ann Linehan and Ross Weaver for their thoughtful comments. Deborah 
Drake, M.A., Early Childhood Education Manager, FHI 360 Region XII Training and 
Technical Assistance provided particularly insightful comments and links to 
additional resources. 


The staff of the Head Start National Center on Program Management and 
Fiscal Operations provided invaluable guidance and feedback from project inception 
to conclusion to assure that these materials aligned with theirs and could 
appropriately supplement existing resources. Perhaps most importantly, we thank 
the eight Head Start sites who participated in the Head Start Leadership, 

Excellence, and Data Systems study (LEADS) project and the five Head Start 
programs whose staff reviewed and provided feedback on an early version of the 
resource guide (four that participated in the study and one that did not). Finally, 
we appreciate the input of other Urban Institute project team members including 
Monica Rohacek, Eleanor Pratt, and Michael Marazzi, and the input of Urban 
Institute senior advisors - Elizabeth Peters and Elizabeth Boris. 


Contents 


Acknowledgements iii 

Overview and Purpose 1 

Organization and How to Use this Resource Guide 3 

Who Should Use This Guide? 3 

How Should You Use This Guide? 3 

PART I: Background 6 

The Changing Compliance and Performance Landscape 6 

Moving from Compliance to Learning in Head Start 8 

Types of Data Head Start Requires 9 

Compliance-only versus Learning in Head Start— What Might That Look Like? 11 

PART 1 1 : The Culture Shift to Continuous improvement 17 

In Practice 18 

Organizational Readiness for Continuous Learning and Improvement 21 

PART III: Using Data for I nternal Program I mprovement 25 

Using Data for Internal Program Improvement 25 

Developing Systems for Continuous Quality Improvement 27 

The Continuous Improvement Cycle 28 

Learning How to Use the Continuous Improvement Cycle 30 

1. Early Math Skill I mprovement for Four-Year-Olds 30 

2. Improving Child-Teacher Interactions 34 

3. Prioritizing Allocations When Budgets Decline 39 

PART IV: Preparing Your Data For Use 44 

Ensuring Data Quality 44 

Validity 44 

Reliability 45 

Accuracy and Completeness 45 

Disaggregating Your Data 48 

Analysis and Interpretation 49 

PART V: Engaging Staff and Stakeholders 52 

Overview 52 

1. I nvolving Staff 53 

2. Engaging your Policy Council, Governing Body, and Key Stakeholders 58 


IV 


3. Encouraging Families 60 

Appendix A: Aligning this Resource Guide with Existing Flead Start National Center Program 
Management and Fiscal Operations (PMFO) Technical Assistance Materials 63 

Appendix B - Continuous Quality Improvement: Conceptual Framework 66 

Appendix C— Flead Start Frameworks for Continuous Program Improvement 69 

Appendix D— Resource List 74 


v 



Overview and Purpose 

Head Start has long focused on assessing and improving program quality to ensure 
that the children served receive the best possible preparation for school and life. 
Most research has been focused inside the classroom - the classroom environment, 
teacher qualifications, and teacher interactions. Of course the classroom is 
important because that is where children spend most of their time, but what is done 
in the classroom isn't entirely or primarily the teacher's decision. The management 
and leadership of the program typically make the decisions about which curriculum 
and assessments to use, the equipment and materials to provide, and how to 
interact with the children. Thus, it is important to study, understand, and provide 
guidance for those managers and leaders in their decisions. 

Accordingly, the Office of Planning, Research and Evaluation (OPRE) 
contracted with the Urban Institute in 2012 to conduct the Head Start Leadership, 
Excellence, and Data Systems (LEADS) project. The LEADS project's goal is to 
understand the factors in organizational and management systems that promote 
effective early childhood education practices and outcomes. The LEADS project has 
three primary products: (1) a literature review and conceptual framework drawing 
from the work of other disciplines that have studied data use for quality 
improvement, 1 (2) documentation of promising practices in Head Start programs 
around data use for continuous quality improvement, 2 and (3) this resource guide 
to translate the research into information community-based Head Start programs 
can use. 

The purpose of this resource guide is to help leadership, management, 
supervisory, and data-focused staff in Head Start and Early Head Start programs 
(1) understand how data, including the data they already collect, can help them 
achieve their program goals; (2) learn techniques for fostering a culture of learning 
in their organization; and (3) increase their ability to identify and address gaps and 
continuously improve their programs. This document provides guidance on 


1 Derrick-Mills, Teresa, Sandstrom H., Pettijohn S., Fyffe S., and KoulishJ. Data Use for 
Continuous Quality Improvement: What the Head Start Field Can Learn From Other 
Disciplines, A Literature Review and Conceptual Framework. OPRE Report # 2014-77 
(Washington, DC: US Department of Health and Human Services, Administration for 
Children and Families, Office of Planning, Research and Evaluation, forthcoming). 

2 Derrick-Mills, Teresa, and Monica Rohacek. Data-I nformed Continuous Improvement in 
Head Start: A Cross-Case Study of Local Grantee Practices (Washington, DC: US 
Department of Health and Human Services, Administration for Children and Families, Office 
of Planning, Research and Evaluation, forthcoming); and Derrick-Mills, Teresa. 
Understanding Data Use for Continuous Quality Improvement in Head Start: Preliminary 
Findings. OPRE Report # 2014-85 (Washington, DC: US Department of Health and Human 
Services, Administration for Children and Families, Office of Planning, Research and 
Evaluation, forthcoming). 
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enhancing data use for quality improvement by drawing upon LEADS project 
research on data use in other fields, promising practices observed in Head Start 
programs, and existing Head Start technical assistance and training materials. 
Fostering data use and a learning culture is hard work; it helps to have multiple 
resources from which to draw. 
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Organi zati on and How to Use thi s Resource Gui de 


Who Should Use This Guide? 

The guide is designed to provide resources, information, and strategies to Head 
Start leaders, managers, and others who support data use in Head Start programs. 
Appropriate users of this guide will already be familiar with Head Start, have some 
knowledge about the types of data that Head Start programs are required to 
collect, and have some understanding of how to use data in a service delivery or 
management setting. 

Technical assistance staff or consultants may find the guide a helpful 
resource for working with Head Start grantees who are trying to build a learning 
culture in their organization, who need some additional examples of ways they 
could be using data to improve program quality, or who want suggestions about 
ways to involve more staff in data use. This guide was designed in consultation with 
staff at the Head Start National Center for Program Management and Fiscal 
Operations (PMFO). In particular, this guide complements the "Using Data 
Effectively: Four Data Activities" workshop materials and the TA Planning Papers. 
See Appendix A for additional guidance on how to use this guide in conjunction with 
PMFO resources developed around the same time period. 

Head Start leaders, managers, and individuals attempting to promote data 
use may find the guide a helpful resource for providing examples and tips to staff 
about what data use can help them accomplish, involving more staff in data use, 
and thinking about how to make time for data use. 


How Should You Use This Guide? 

For most users, we recommend that you read or at least skim through the entire 
guide to decide which parts are most useful to your organization or situation. Within 
the guide there are many tips and resources that you might choose to pull out for a 
particular conversation or discussion. The tips, resources, and examples could be 
used separately, but the people introducing them to others should have the 
background the guide provides to be sure they are introducing the concepts 
correctly. 

This resource guide is organized into five parts (the sub- bullets are used to 
"call-out" specific resources, examples, or exhibits) in each part: 
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• Part I describes the shift from an emphasis on compliance to an emphasis 
on performance within government- funded programs. It offers definitions of 
key terms used throughout the resource guide so that all Head Start staff 
can build a common vocabulary around data use for program performance. 
It also situates data use in the context of Head Start. 

o Required Types of Head Start Data Collection and Data Type 
(Page 9) 

o Compliance-only vs. Learning & A Hypothetical Data example 
(Page 11) 

• Part 1 1 introduces readers to new ways of thinking about organizational 
culture based in continuous improvement. This part stresses the importance 
of curiosity, reflection, and trust among staff at all levels. It also describes 
the steps to embarking on continuous improvement and the core 
competencies that define it. 

o I ndicators of a Culture of Continuous I mprovement (Page 18) 
o Fighting Fires to I nnovation: An Analogy for Learning (Page 
20 ) 

o Core Competencies of Organizations with a Culture of 
Continuous Improvement (Page 23) 

• Part 1 1 1 describes the cyclical practice of data collection, analysis, and use 
for internal improvement. It connects the program planning for Head Start 
cycle with a more detailed description of the steps necessary to improve 
every day. Step-by-step examples of data- informed program learning, 
including efforts to improve 4-year-olds' early math skills, teacher-child 
interactions, and program budgeting, are included. 

o Using Data for I nternal Program I mprovement (Page 26) 
o Program Planning Cycle (Page 27) 
o Continuous Improvement Cycle (Page 28) 

o Scenario 1: Early Math Skills I mprovement for Four-Year Olds 
(Page 30) 

o Scenario 2: Improving Child-Teacher Interactions (Page 34) 
o Scenario 3: Prioritizing Allocations When Budgets Decline 
(Page 39) 

• Part IV provides an introduction to assuring that your data are ready for 
meaningful use, including by attending to such issues of data quality as 
validity, reliability, accuracy, and completeness. This section also provides 
tips for data disaggregation. 

o Elements of Data Quality Tip Sheet (Page 47) 
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o Client and Service Characteristics for Use in Disaggregating 
Data (Page 48) 

o Using Data to I m prove Quality (Page 50) 

• Part V, in contrast to the first four parts of this resource guide that focus on 
the work of Head Start staff, explains the importance of and steps for 
involving parents, teachers, Policy Council members, and other stakeholders. 
It includes tips on incorporating data into regular staff meetings, board 
meetings, and parent conferences. It also outlines "data walks," a new 
technique for fostering conversation and joint problem solving. 

o The Four A's of Effective Reports: Appealing. Accessible. 
Accurate. Audience-specific (Page 59) 

For more information, turn the page to begin Part I, or jump ahead to these items 
of interest: 

• Types of data. See page 9. 

• Differences between compliance and learning. See page 11. 

• Indicators of a culture of continuous improvement. See page 18. 

• Your organization's readiness for continuous learning. See page 21. 

• Seven wavs to use data for internal improvement. See page 26. 

• Developing a continuous cycle of improvement. See page 27. 

• Specific examples of continuous cycles of improvement. See page 30. 

• How data can help you make hard budgeting decisions. See page 39. 

• How to prepare vour data for use. See page 44. 

• Dimensions for disaggregating vour data. See page 48. 

• Involving staff more in data use. See page 53. 

• Engage other stakeholders in data use. See page 58. 
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Whether it is called performance management, data- informed decision-making, 
or continuous quality improvement, the goal is for Head Start Programs to use 
the data they are collecting to make decisions, especially decisions that will help 
them improve quality and thus achieve better outcomes for the children, 
families, and communities they serve. 


PART I : Background 


The Changing Compliance and Performance Landscape 

In any government-funded program, accountability to citizens on the use of funds is 
important. All government programs have rules regarding spending and record- 
keeping that are designed to prevent waste, fraud, and abuse. Essentially, these 
are requirements that assure the money is used as intended when it was authorized 
by Congress (if a federal program) and disbursed by the government agency. 
Everyone wants assurances that the money they pay in taxes is not being misused; 
this requires compliance-accountability. 3 

The types of requirements and the expectations of what it means to be a 
good steward of the funds have changed over time. Though citizens still expect 
government to prevent the misuse of government funds, many now expect 
government to show what it has accomplished, such as how many individuals it 
served (efforts) and how the lives of those individuals were changed (outcomes). 
This newer type of accountability focuses on performance. It is often referred to as 
results- based accountability or results-oriented management. This performance- 
based accountability was systematically introduced at the federal level in 1993 
through the National Performance Review and through Congressional passage of 
the Government Performance and Results Act. 

The idea behind these performance measurement systems was to improve 
program outcomes and quality. The systems originally assumed that if performance 
information was being reported out, then it would also be used internally to 


3 Callahan, K. Elements of Effective Governance: Measurement, Accountability and 
Participation (Boca Raton, FL: Auerbach Publications, Taylor & Francis Group, 2007). 
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improve performance. Research on the performance measurement efforts, 
however, found that organizations were complying with the requirements to collect 
and report on the performance measures, but few were using the data internally to 
make program improvements. These findings were consistent across levels of 
government and in community- based organizations to which government funds 
were awarded to implement the government programs. Successive federal 
administrations initiated their own performance measurement strategies, and 
Congress updated the Government Performance and Results Act in 2010. Each 
subsequent iteration increasingly emphasized not only measuring performance but 
also managing to achieve results. 4 In nonprofit and public management, managing 
to achieve results is called performance management; in the education field it is 
referred to as data-driven decision-making; 5 and in healthcare it is called 
continuous quality improvement. 6 


? 

Compliance 

Performance 

We followed all 

Look at what we 

the rules. 

accomplished! 


Whether it is called performance management, data- informed decision- 
making, or continuous quality improvement, the goal is for the organization to use 
the data it is collecting to make decisions, especially decisions that will help 
improve program quality and thus achieve better outcomes for the individuals 
served. In government agencies that allocate their funds to other organizations to 
provide services, the hope is that learning will occur at all levels. Ideally, the 
federal agency would use the data provided to them to learn about how to support 
program improvements, and the organizations receiving the federal funds would 
learn from the data they collect to improve the services they are providing. 


4 Wholey, J . "Managing for Results: Roles for Evaluators in a New Management Era." 
American J ournal of Evaluation 22 no. 3 (2001): 343-47. 

http://www.classtoolkit.org/sites/default/files/documents/Managing_for_Results.pdf. 

5 Goren, Paul. "Data, Data, and More data— What's an Educator to Do?" American J ournal of 
Education 118 no. 2 (2012): 233-37. 

6 Blumenthal, David, and Charles M. Kilo. "A Report Card on Continuous Quality 
Improvement." Milbank Quarterly 76 no. 4 (1998): 625-48. 
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2751093/pdf/milq_108.pdf. 
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Research shows that organizational learning is not easy even when 
organizations want to do it. The companion Literature Review 7 explores 
multidisciplinary research on the challenges and facilitators of organizational 
learning and presents a conceptual framework showing the elements that support 
and impede organizational learning (See Appendix B). This resource guide focuses 
on how to develop an organizational culture that supports data use and learning. 


Moving from Compliance to Learning in Head Start 

As a government program, Head Start experienced the early public management 
waves around external accountability and data collection for compliance. The 
Program Information Report (PIR) that all Head Start agencies are required to 
populate with data and provide to the Office of Head Start is representative of a 
typical requirement from that era. Similarly, guidance to share management 
reports with the governing body and Policy Council are focused on transparency of 
actions and accountability. 8 The Head Start Act of 2007 requires the sharing of such 
items as the Self-Assessment, community assessment, USDA Child and Adult Care 
Food Program reports, and enrollment reports. 

The implementation of the Head Start Designation Renewal System (DRS) is 
a new wave in Head Start's own data reform movement. The DRS and the 2007 
reauthorization of Head Start place an increased emphasis on ongoing assessments 
of children and the use of data about children's school readiness for program 
improvement. Just as organizations in other fields have worked to balance data 
requirements for compliance and external reporting with organizational learning, 
Head Start organizations are being compelled to do the same. 

The Advisory Committee on Head Start Research and Evaluation 9 laid out two 
goals that are relevant here. Head Start should (1) become a learning organization 
(from the federal level down through local, community-based organizations) "where 
decisions about instructional practices and curricula, assessments, monitoring, 
professional development, and organizational development are integrally linked to 
each other and to school readiness and other key goals"; and (2) "expand the 


7 Derrick-Mills et al., Data Use for Continuous Quality Improvement. 

8 "Program Governance," National Center on Program Management and Fiscal Operations, 
accessed December 8, 2014, http://eclkc.ohs.acf.hhs.gov/hslc/tta- 
system/operations/mang-sys/ program- gov/ program-governance.html. 

9 Advisory Committee on Head Start Research and Evaluation. Final Report (Washington, 
DC: US Department of Health and Human Services, 2012). 
http://www.acf.hhs.gov/sites/default/files/opre/eval_final.pdf. 
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evidence base where it is insufficient, and to rely on existing evidence from 
research, evaluation, and ongoing monitoring of progress to develop and 
continually refine programs to ensure that they are systematic, intentional, and 
intensive enough to achieve their goals for children's school readiness and family 
well-being." 

Interviews with Head Start programs 10 suggest that many of them are trying 
to become learning organizations. They are examining the different kinds of data 
they are required to collect to determine what they can learn from the data rather 
than viewing them only as data that fulfill a requirement. Additionally, rather than 
just presenting the required reports to their governing bodies and Policy Council, 
some Head Start programs are discussing the reports with them and including them 
in the process of deciding what the reports mean and how to handle the strengths 
and opportunities identified in the reports. 

Below we describe the types of data that Head Start programs are required 
to collect and categorize the types into inputs, efforts, process, and effects or 
outcomes. 


Types of Data Head Start Requires 

Head Start programs are required to collect many types of data through several 
defined data collection processes. The data collection processes include the 
community assessment, ongoing record keeping, Self-Assessment, and ERSEA 
(Eligibility, Recruitment, Selection, Enrollment, and Attendance). Head Start 
programs are required to report some of their data through (1) the PIR, which 
stipulates which fields they must collect; (2) ongoing monitoring reports; and (3) 
an annual report. When monitored, grantees must demonstrate they are collecting 
the appropriate data and sharing it when and where required. 


10 The Head Start Leadership, Excellence, and Data Systems Project conducted interviews 
with staff at eight Head Start programs around the country in spring 2014. 
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Table A. Head Start Data Collection Requirements and Data Type 


Data 

1 nputs 


Effects or 
outcomes 

collection 

required 

Community 

Children 

and 

families 

Staff 

Efforts 11 

Community 

assessment 

X 

X 




Record- 

keeping 



X 

X 

X 

Self- 

assessment 


X 

X 


X 

ERSEA 


X 


X 



Source: This table has been constructed by the aut 


hors to represent the types of data 


available through many of the required Head Start data collection routines. 


Note: Tracking of financials is also required, but that is not this table's focus. 


Performance measurement and other continuous quality improvement fields 
typically divide data into types corresponding to stages of the service delivery 
process. In Table A, we categorize the types of data Head Start programs are 
required to collect into inputs, efforts (or outputs), and outcomes (or effects). We 
briefly define these types of data below. When available, we use definitions found in 
other Head Start- oriented materials, but we point out other common terms in case 
your program is using information from many sources to help you in meeting your 
data and analysis needs. 

• Inputs are the human, financial, organizational, and community resources that 
the organization accesses in implementing the project. 12 Budget resources are 
not included in this chart but are important. In Head Start, the types of children 
and families served, the community in which services are provided, and the staff 
providing the services are all important inputs or resources to consider in service 
design and delivery. 


11 In some professional fields, the term "outputs” would be more common than "efforts." 

12 W.K. Kellogg Foundation. Using Logic Models to Bring Together Planning, Evaluation, and 
Action: Logic Model Development Guide (Battle Creek, Ml: W. K. Kellogg Foundation, 2004) 
http://www.wkkf.org/resource-directorv/ resource/ 2006/ 02/ wk-kellogg-foundation-loaic- 
model-development-auide . 
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• Efforts (or outputs) are what and how much your program does and how well it 
does it. 13 For example, the numbers of children and families served, the 
numbers of home visits performed, the number of parent workshops held, and 
the quality of those services are all efforts. The services your program offers, 
and how well you perform them, lay the foundation for possible outcomes. The 
phrase "how well" refers to how quickly, how routinely, and with what level of 
quality something is accomplished. For example, you may have a home visitor 
who is visiting families in a timely manner, but not really connecting with the 
family. This service is timely but not of the level of quality you expect. Tracking 
and reflecting on your efforts is an important step in understanding how you can 
improve on your outcomes. 

• Outcomes (or effects) are "something that happened as a result of an activity 
or process. The actual results achieved. The term outcome is also used to refer 
to expected outcomes. That is, the results you expect to see because of an 
activity or process ." 14 Outcomes may also be called effects, or "measures of 
changes in knowledge, attitudes or behaviors ." 15 Outcomes are not the same as 
impacts. "Impact" is a term used to signify results that have been verified or 
tested through an experimental or quasi-experimental research study . 16 The 
primary difference between outcomes and impacts is that outcomes measure the 
changes you expect to see over time, but it's difficult to determine how much of 
that change is a result of the actions and experiences in your program rather 
than the experiences and services that families and children would have without 
your program. Measuring outcomes, however, determines whether children and 
families are making the expected progress and provides guidance on how to 
facilitate that progress if they are not making it. 

Compliance-only versus Learning in Head Start— What 
Might That Look Like? 

Head Start programs have many data-related requirements. This can create both 
opportunities and challenges to use of data for learning. If organizations are 
collecting many types of data already, they may already have management 
information systems to help organize the information. However, they may be used 
to thinking about these data sources only as requirements rather than as 


13 PMFO. Foundations for Excellence: Planning in Head Start (Washington, DC: US 
Department of Health and Human Services, Administration for Children and Families, Office 
of Head Start, 2014), 4. http://eclkc.ohs.acf.hhs.gov/hslc/tta- 

system/operations/ planning.html. 

14 PMFO, Foundations for Excellence, 4. 

15 PMFO, Foundations for Excellence, 4. 

16 PMFO, Foundations for Excellence, 4. 
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information for improvement. So what might data collection for compliance look like 
as compared with data collection for learning? A possible scenario follows. 

The Scenario 

For many children, school readiness requires support from disability services. These 
services are delivered by both mainstream and special education staff and tailored 
to the needs of each child through an individualized family service plan (IFSP, for 
children younger than 3) or individualized education plan (IEP, for children ages 3 
to 21). Research shows that early identification and intervention are important for 
children's success in Flead Start and beyond. Accordingly, disability services are an 
important part of the Head Start philosophy and requirements. In this section, we 
outline how data collection and reflection on these disability services may look 
different in organizations that are operating primarily under a compliance culture, 
are transitioning to a learning culture, or are fully developed as learning 
organizations. 

Head Start Performance Standards require that programs 

• collect and report data through the PIR on (1) the number of children 
enrolled in the program who had an IEP or IFSP at any point during the year, 
whether they were determined eligible before enrollment or during the 
enrollment year, and (2) the number who have not received special 
education or early intervention services; and 

• complete certain screenings and referrals within a specified period after 
enrollment. 

The following text box illustrates a few ways in which data may be used to 
support different uses within an organization. 
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Compliance 

The program collects and reports the required data 
through the PIR. The program tracks and records the 
timeliness of screenings and referrals. The program 
performs all elements within the specified periods, 
collects all the required data and maintains all the 
required records, and reports on the data when 
required. 

Transitioning 

The program does all of the required items for the 


Learning 

The program does all of the required items for the 
"compliance" descriptor and performs the types of 
analyses indicated in the "transitioning" descriptor, but 
it looks at data by types of children rather than solely 
by child. Program staff have heard about some research 
showing that certain types of children tend to be over- 
or underidentified for disability services. Staff regularly 
monitor data about children with and without lEPs and 
IFSPs for signs of disproportionality. 3 If they see signs 
of disproportionality, they will investigate possible 
reasons. Staff disaggregate their data (break it into 
categories) as shown in Table B and organize teams to 
interpret the data. 


a Disproportionality is defined as the over- or underrepresentation of certain groups of 
children (e.g., racial or ethnic groups, genders, or groups by home language) within specific 
categories of ability or disability. 
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Table B. Hypothetical Example of 

Column 1 

. . .. Children with 

Characteristic 

an FSP 

Data Disaggregation: Use of Disability Services 

Column 2 Column 3 Column 4 

Children with Children with 

an 1 FSP not Children with an 1 EP not 

receiving an 1 EP receiving 

services services 

Gender 





Female 

12 

4 

12 

5 

Male 

18 

2 

33 

3 

Age group 





1-year-olds 

16 

3 



2-year-olds 

14 

3 



3-year-olds 



21 

7 

4-year-olds 



24 

1 

Race or ethnicity 





African American 

15 

2 

23 

2 

Asian 

0 

0 

2 

2 

Hispanic 

5 

2 

3 

2 

White 

10 

2 

17 

2 

Home language 





Language other 

7 

0 

4 

0 

than English 





English only 

23 

6 

41 

8 

Total 

30 

6 

45 

8 


The data presented in Table B are already collected as part of the PIR Head 
Start reporting requirements. These data, however, can be used in other ways to 
help Early Head Start and Head Start programs better understand characteristics of 
children enrolled in services. This table disaggregates data at a hypothetical large 
program with both Early Head Start and Head Start services. In this hypothetical 
program, 75 children qualify for disability services. Thirty of these children are 
enrolled in Early Head Start; 45 are enrolled in Head Start. Columns 1 and 3 show 
the population of children who qualify for special education services. Several 
patterns emerge that may be of interest to the program. 

• Many more boys than girls have an IFSP or IEP (18 boys compared with 12 
girls and 33 boys compared with 12 girls, respectively). 

• Slightly more 1-year-olds qualify for disability services than 2-year-olds (16 
compared with 14, respectively), and slightly more 4-year-olds qualify for 
disability services than 3-year-olds (24 compared with 21, respectively). 
Third, examining race or ethnicity, the greatest number of children with 
IFSPs or I EPS are African-American (38 total); far fewer white (27 total), 
Hispanic (8 total), and Asian (2 total) children qualify for disability services. 

• Few of the children with IFSPs and lEPs have a home language other than 
English (11 total); this follows national trends. 
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• In columns 2 and 4 of Table B, the numbers of children with IFSPs and lEPs 
who are not receiving services are shown. This is another area where data 
disaggregation can illuminate differences between groups and inform 
changes in policies or procedures. Although this analysis found that more 
boys than girls had IFSPs and lEPs, here the data show that more girls than 
boys are not receiving services. 

• More 3-year-olds than 4-year-olds are not receiving special education 
services (7 compared with 1, respectively). In contrast, although different 
numbers of children from each racial/ ethnic group have lEPs, the same 
number of qualifying children from each group is not receiving services (2). 

• All children with IFSPs and lEPs whose home language is something other 
than English are receiving disability services; 14 children whose home 
language is English are not receiving these services. 

Data disaggregation is most useful when paired with careful interpretation. 
Thus, Table B is an example of what the organization with a learning culture may 
have developed for their data interpretation teams to reflect on the data, 
investigate possible reasons for observed differences, and make suggestions about 
internal policies, procedures, and partner activities that may need to change. 
Regarding use of disability services, Table B brings up the following questions: 

• Do differences in IEP qualification by gender and race or ethnicity reflect 
differences in the population of children enrolled at the Head Start program? 

• Do differences in IEP qualification suggest 

o an overrepresentation of boys in special education services, 
o an overrepresentation of African American children in special education 
services, or 

o an underrepresentation of children whose home language is not 
English in special education services? 

• Why are some children not receiving disability services? 

• How are special education resources allocated to children and families of 
different groups? 

• How can teachers and special education staff ensure that all children meet 
the school- readiness goals set by Head Start? What changes in disability 
services screening, referrals, or support may be needed to do so? 

After looking at the data on disability services use by child and family 

characteristics, the data review team determines they need more information. 

They are not sure that the differences are really because of child characteristics. 

The differences may arise because of different contracted service providers, the 

availability of services in the neighborhoods where the children live, or some 
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families' cultural beliefs. They plan to investigate further by disaggregating 
based on additional characteristics, such as those found in Table C in Part IV. 
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PART 1 1 


The Culture Shi ft to Continuous improvement 


Organizations seek a culture of continuous improvement and aspire to high 
performance by examining all levels of the organization rather than monitoring and 
evaluating in isolation. Of course, monitoring, evaluation, compliance, and 
accountability are important because you must do what you were funded to do. As 
you establish a culture of learning, however, your staff think beyond requirements 
and aspire to do more than expected and look to data as a source of information for 
improvement. 

As programs and organizations shift their thinking from mere compliance to 
continuous inquiry, they quickly encounter a host of terms and conceptual 
frameworks (see Appendix C for some of the frameworks currently suggested in 
Head Start resources). Supplementing compliance and accountability are such 
terms as continuous inquiry, continuous learning and improvement, and learning 
culture. Head Start programs have a built-in mechanism to foster a culture of 
inquiry: their ongoing monitoring system. 

A culture of continuous improvement is generally synonymous with learning 
culture or continuous inquiry. The term "continuous” suggests a cycle or a feedback 
loop that repeatedly challenges individuals to consider what is working and what 
can be done differently to improve or achieve better results. This process of inquiry 
is continuous rather than episodic. Individuals in continuous improvement cultures 
are always asking questions and seeking answers to those questions. Organizations 
actively pursuing a culture of continuous improvement create a safe space for staff 
and stakeholders to ask, reflect, and think more creatively about solutions. 
Foremost, the organization must cultivate a sense of trust throughout the agency. 
Those in executive and management roles must model a curious yet humble spirit. 
Some of these characteristics are elaborated on in the text box below. 

Creating a culture responsive to this style of learning is arguably the most 
significant factor in making the shift from a culture solely focused on compliance to 
one focused on continuous improvement. 
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I ndicators of a Culture of Continuous I mprovement 


• Curiosity — asking the "how” and "why" questions about children and 
families in your program 

■ Reflection — continuously reviewing program policies and seeking 
feedback, applying when necessary 

■ Tolerance of failure and vulnerability — recognizing when things aren't 
working and making appropriate course corrections 

■ Use of feedback — using data to assess whether or not strategies or 
programs are making a difference for children and families; this is similar 
to Reflection 

■ Systems thinking — stepping back and considering the broader context in 
which Head Start programs operate and understanding that change is 
incremental. 

Source: National Center on Quality Teaching and Learning. 2011. School Readiness for All Children: Using 
Data to Support Child Outcomes. Washington, DC: US Department of Health and Human Services, 
Administration for Children and Families, Office of Head Start. 

http://www.hitinc.org/uploads%5Cresources%5C1308%5Cncqtl data handbook version 11 1 11. pdf . 


I n Practice 

In practice, most organizations, including Head Start programs, need to 
demonstrate they are complying with requirements while building a culture of 
continuous improvement. Though it can be a challenge to do both, the functions, 
actions, and data are similar. Individuals in society frequently follow rules, but they 
often add their own learning to the situation. For example, consider the rule that 
pedestrians in a crosswalk have the right of way. Though an individual pedestrian 
knows he or she could legally step out in front of car moving too quickly to stop, he 
or she would also have learned through observations that he or she is likely to be 
injured by doing so. Individuals modify their behaviors every day to improve their 
safety and quality of life. They observe their surroundings, read crime reports and 
newspapers, view news programs, and learn about their environment in many other 
ways. 


Applying those principles to an organization requires the creation of systems 
because, though we want each individual to comply and to learn, we need a 
coordinated effort across individuals, departments, and sites. We also need to be 
complying and learning across multiple dimensions. We need to create systems for 
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observation, reflection, learning, and action. We need systems that help us identify 
and solve problems proactively instead of always reacting. 

The illustration on page 20 (excerpted from Baldrige Performance Excellence 
Program's "Education Criteria for Performance Excellence,") shows how 
organizations can progress from the earliest stages of problem solving to more 
advanced stages of organization-wide improvement and innovation. According to 
the Baldrige criteria, early-stage organizational operations are characterized by 
"activities rather than by processes, and they are largely responsive to immediate 
needs or problems." The most advanced or mature organizations exhibit processes 
that are repeatable and in collaboration with other units. These organizations are 
also "regularly evaluated for change and improvement." 

Of course, knowing how to fight fires is an important skill. You will always 
have some emergencies that need immediate attention. The goal, however, is to 
evaluate how to allocate limited resources. If you are constantly fighting fires, you 
don't have much time or energy for reflection. Shifting from fighting fires constantly 
to preventing fires can be a hard transition, but once a system is in place, keeping 
up prevention and improvement activities becomes routine. In what areas is your 
organization constantly fighting fires rather than learning? 
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From Fighting Fires to Innovation: An Analogy for Learning 


Learning essential attribute of high- 
petforming organization. Effective, weSf-deployed 
orpnFzatlpn:i I learning can helps an organization 
Improve from the early stages of reacting lo 
problems to the highest levels of wganizatfon- 
vvirle Imjj&vement, resTnemenb and innovation. 



Reading lit the problem 

Run wlh tJ^ hose and pul nut the fire, 



General impnuvcmenl oriental ion 

Install mofli T-re hoses to gd To fte bras qulddy 

iirt d mucL: Mvi r in:p'uiJ. 



^lEmalic evaluation and improvement 

evaluate which locations a*e mast siaceptible Da fira. Install 

twin! iSrtschrs jni I ■. p :r •-nk'iiir^ in lhciKt± lociOkmn. 



Learning And stralegic improvement 

SnE-tall sy&TemwfdE heat sensors and a sprinkler system 
ih^i Is jclivaied by lht hem preceding fires. 



Organizational analysis* and innavalEon 

Use fireproof and fire-nQlardant snaTennls. Replace combiMtible 
NquIrjV; wliU walei-baseJ [Iqutds. JVev*nl:on is ihe primary 
approach be protection; wilh sensors and sprinklers as the 
secondary line of prtMeelion. 


F.-dti A/dr-gt ^rrSa-nirfrt* hiLrlli .-c f Pl^rjn,. 201 I, Ml-i-JO 14 frLcm (a 1 rac r. lilc vTij, C-l tL:L'^ mH 1 L..S. HcLurlfrcr^ 

CiHfrarct, N-a i i d n al I n n :Y^x -pf S tWa m r and fed^o^ h Up r^J. pewft a I d r- g tp u b 1 1 cjtrc^tiu f r*? r_n p r ro-f fn_ m I enj . c 1 ’ v 


Source: "Baldrige Performance Excellence Program: Education Criteria for Performance 
Excellence." Baldrige Performance Excellence Program, accessed November 5, 2014, 
http://www.baldriae.nist.gov/PowerPoint Files/2012 Education Criteria.ppt . 
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Imagine a scenario where one of your teachers comes to your Education 
Coordinator (Ed Coordinator) for help. She has a child in her class who is having a 
hard time regulating behavior. The disruptive behaviors happen frequently, she 
feels like she spends most of her time managing this one child's behavior, and she 
needs new strategies. In this example, the teacher recognizes that she is fighting 
fires in the classroom. The Ed Coordinator provides ideas on how to diffuse the 
situation once it starts (a better way to put out the fire). The Ed Coordinator also 
provides coaching to the teacher about how to identify triggers for the behavior and 
how to structure her classroom and activities to avoid those triggers (preventing 
the fire). Did your Ed Coordinator put out this fire and say "I'm done addressing 
this issue," or did she think about how to investigate whether this fire is a symptom 
of something bigger that should be addressed more broadly? This teacher asked for 
help, but are there others that didn't? Could your organization prevent future 
incidents like this with more orientation or training? We address these questions in 
the next section. 

Organizational Readiness for Continuous Learning and 

I MPROVEMENT 

How do you know how prepared your organization is to engage in continuous 
learning and improvement? One way is to see where you fit along a continuum of 
behaviors, actions, and conditions that exemplify different stages of readiness. 
Although initially written to help nonprofits think about social media networks and 
strategies, the following developmental phases (adapted from Beth Kanter's blog 
post "How Can Nonprofits Switch to a Data- 1 nformed Culture?”) 17 are likely 
applicable to several stages of organizational readiness in Head Start programs. 

• Dormant: At this stage, the organization does not know where to start. Data 
collection may occur from time to time, but there is no formal reporting. 
There are no data systems in place, such as dashboards or simple collection 
methods. Staff are often overwhelmed by the thought of measurement and it 
falls to the bottom of the to-do list. Alternatively, there may bean emphasis 
on collecting more data than is necessary, but no one relates it to decision- 
making. There is not a reflection process for analyzing success or failure for 
future use. 

• Testing and Coordinating: At this stage, the organization is regularly 
collecting data, but it is stored across different spreadsheets and collected by 
different people or departments. Data are not linked to organizational results 
or mission-driven goals across programs. Discussions on how to improve 
results are rarely part of staff meetings. 


17 Kanter, Beth. "How Can Nonprofits Switch to a Data-I nformed Culture?" Beth's Blog, 
August 17, 2011. http://www.bethkanter.org/switch-data-driven/ 
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• Scaling and Institutionalization: At this stage, there is an organization-wide 
system and dashboard for collecting data that are shared with different 
departments. There are different views or levels of detail for senior leaders, 
line staff, or other stakeholders. There are periodic (e.g., weekly, biweekly, 
monthly, or quarterly) check-ins to evaluate what is working and what is not. 
The organization provides training and professional development for staff to 
learn how to use measurement tools. 

• Empowering At this stage, performance indicators are used across programs 
throughout the organization. There is a staff position responsible for setting 
the overall agenda for data collection and reporting, helping staff understand 
data, and assuring that systems and timelines are successful. All staff, 
however, are empowered and expected to check, apply, and interpret their 
own data. In addition to periodic check-ins, the organizational dashboard 
includes goal-oriented performance metrics. The organizational dashboard is 
shared across departments and there is a process for analyzing, discussing, 
and applying results. Data visualization techniques are used not only to 
report the data analysis but also to reflect on best practices culled from the 
data. 

It may be that you can see your organization in one or more of these stages 
at any given time. For example, you might be overwhelmed with many data 
reporting requirements (dormant) or have data in multiple systems or spreadsheets 
(testing and coordinating). Simultaneously, you might be having regular check-ins 
(scaling and institutionalization) to make sense of the information you have. The 
goal is for your organization to become aware of what it means to be a data- 
informed, learning organization and to integrate better-aligned, coordinated 
activities and systems at all levels of the organization. 

The descriptive statements in the following text box offer guidance to help 
you think about your current readiness and suggest specific ways you can help to 
advance your program along this continuum. 
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Core Competencies of Organizations 
With a Culture of Continuous I mprovement 

> Our organization measures outcomes (changes in participant condition, 
behavior or knowledge), not just efforts (quantifiable activities or services 
delivered). 

> Our organization can identify which indicators are appropriate for measuring 
how we work. 

> Our organization has clarity about what we want to accomplish in the short 
term (e.g., one to five years) and what success will look like. 

> Our organization ensures that staff have the information and skills they need 
to successfully engage with data for program improvement (e.g., access to 
resources and training) 

> Our organization has staff who are experienced in data collection, data use, 
and different stakeholders' information needs. 

> Our organization has staff who know how to analyze data and interpret what 
the data mean. 

> Our organization values learning. This is demonstrated by staff actively 
asking questions, gathering information, and thinking critically about how to 
improve their work. 

> Leaders in our organization support data use to identify areas of 
improvement. 

> Our organization is capable of effectively communicating about data and 
results (both positive and negative) within and outside the organization. 

> Our organization promotes and facilitates internal staff members' learning 
and reflection in meaningful ways regarding data use, planning, 
implementation and discussion of findings ("learning by doing"). 

> Our organization modifies its course of action based on findings from program 
data. 

> Mangers look at program data as an important input to help them improve 
staff performance and manage for results. 

> Findings from program data are integrated into decision-making when 
deciding which policy options and strategies to pursue. 

Source: "Evaluation Capacity Diagnostic Tool." Informing Change, accessed November 5, 2014, 
http://informinqchanqe.com/uploads/2010/06/Evaluation-Capacitv-Diaqnostic-Tool.pdf . 


For most Head Start programs, moving along these stages of development 
will likely take many years. How quickly a program or organization progresses will 
depend on several factors, including the strength and commitment of leadership, 


23 


the availability of resources for technology and professional development, and the 
willingness of staff to embrace new approaches to data use. 


HIGHLIGHTED RESOURCE 
The District and Data Team Toolkit 


Geier, Robb, and Stephen C. Smith. 2012. District and School Data Team Toolkit. Everett, 
WA: Washington Office of Superintendent of Public Instruction, Washington School 
Information Processing Cooperative, and Public Consulting Group. 
http: // www.kl2.wa.us/CEDARS/Data/pubdocs/ FullToolkit.pdf . 

This toolkit, developed for school districts in Washington state, includes several 
strategies and exercises to engage education administrators and staff in more 
effective data use. It addresses such topics as barriers to effective data use, how to 
manage change and understand staff concerns, and how to create data inventories. 
It also includes many checklists to help organizations reflect on such issues as data 
capacity, data culture, and data usefulness, all of which are easily adaptable to 
Head Start programs. 

Additional resources and exercises can be found on the ESD113 Early Data Learning 
Institute web page. 18 


18 "Early Learning Data Institute," Capital Region ESD 113, accessed November 5, 2014, 
http://www.esdll3.org/Page/1529. 
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PART 1 1 1 


Usi ng Data for I nternal Program I mprovement 


Teachable moments: When we think about opportunities for children to learn, 
we see every moment as holding the potential for learning. We can use their 
daily activities and routines to help them understand new things about the world 
around them and to change their behaviors. 

Continuous quality improvement applies the same logic to organizational 
learning. If you are observing and recording, you will find learnable moments 
every day that you can take action on to improve your organizations. 


Head Start programs are encouraged to continually strive for higher quality, leading 
to better outcomes for children and families throughout their lives. Continuous 
quality improvement requires continuous gathering of information to understand 
new definitions of quality, appropriate indicators for determining whether your 
program is meeting quality benchmarks, and a process for regularly examining 
program management and service delivery to assure they are supporting quality. 

Using Data for Internal Program Improvement 

Using data internally provides many opportunities for program improvement, 
including identification of (1) outcomes for children, parents, or staff that need 
attention; (2) service policies and procedures that need improvement; (3) 
opportunities to improve service delivery and identify successful practices and 
achievements; (4) staff's technical assistance and training needs; and (5) budget 
allocation priorities. 19 Although listed separately, these types of data uses are 
frequently looked at in combination. The following tip sheet, outlining seven 
different ways that data can help you improve your program, is an resource that 
can easily be shared with staff. 


19 Morley, Elaine, and Linda M. Lampkin. Using Outcome Information: Making Data Pay Off. 
Series on Outcome Management for Nonprofit Organizations. Washington, DC: Urban 
Institute, 2004. http://www.urban.org/UploadedPDF/311040 Outcomel nformation.pdf . 
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Using Data for I nternal Program I mprovement 

1. I dentify outcomes that need attention 

In Head Start, the ultimate outcome is how ready children are for kindergarten, but other 
intermediate outcomes provide indicators to let programs know if they are headed in the 
right direction, including attendance rates, rates of skill and knowledge development and 
improvement during the school year, parental participation and satisfaction, and parental 
interactions with their children. 

2. I dentify client groups that need attention 

Clients in Head Start are children and their families. Children and families may differ based 
on the communities where they live, family characteristics (e.g., income, parental 
education, language spoken at home), child characteristics (e.g., age, gender, or type of 
special need), or program participation characteristics (e.g., center-based, home-based, 
part-day or full-day). 

3. Identify service procedures and policies that need improvement 

These could include how and how often different staff interact with parents; the timing of, 
locations of, and process by which applications for enrollment are accepted; prioritization 
of applicants on a waiting list; the frequency and process of child assessments; and 
expectations of how and when to use data. 

4. Identify possible improvements for service delivery 

Examples of service delivery elements include how and where services are provided (e.g., 
home-based or center- based), adding Early Head Start or Head Start program, curriculum 
and assessments used, involvement of parents, and coordination with other community 
programs. 

5. Identify successful practices and achievement 

Managerial observations or analysis of data may reveal particular practices that stand out 
as more effective than others and could be used more widely across the program. 

6. Identify staff's technical assistance and training needs 

Observations and assessments of staff performance and examination of each staff 
person's accomplishments reveal professional development needs. 

7. Determine budget allocation priorities 

When resources are cut or new needs develop, data about what is and isn't working or 
being fully utilized suggest where budgets should be prioritized. 
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Developing Systems for Continuous Quality Improvement 

This continuous process of gathering, reflection, and adjustment to improve quality 
occurs at many levels across several kinds of successful learning organizations 
delivering many different services . 20 The Program Planning Cycle in Head Start is 
designed to help Head Start programs envision a process that incorporates required 
data collection efforts, beginning with the community assessment and culminating 
in a Self-Assessment reflective process. As described in the PMFO "Planning in Head 
Start” resource, the program planning cycle focuses on overall program goals. 
Within those goals are several objectives that break the goals into achievable parts. 


Program Planning Cycle 



Source: National Center on Program Management and Fiscal Operations, n.d. Program 
Planning in Head Start: The Program Planning Cycle. Washington, DC: US Department of 
Health and Human Services, Administration for Children and Families, Office of Head Start. 
https://eclkc.ohs.acf.hhs.aov/hslc/tta-svstem/operations/docs/plannina-cvcle.pdf . 

Here we present a continuous improvement cycle that provides greater detail 
on the steps happening within an organization carrying out the Program Planning 
Cycle and the similar cycles needed to develop and reflect on specific objectives. 
We focus our discussion and examples at the level of objectives, but the cycle 
presented here can be used at any level of goal or objective-setting, review, 
assessment, and adjustment. 


20 Derrick-Mills et al. , Data Use for Continuous Quality Improvement. 
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The Continuous Improvement Cycle 

Successful data use for program improvement requires systems that allow for 
continuous examination of the data, engagement of key staff and stakeholders, and 
the ability to make changes when the needs are identified as illustrated in the 
continuous improvement cycle (Figure A). 


FIGURE A 
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Source: Derrick-Mills, Teresa, Sandstrom, H., Pettijohn, S., Fyffe, S., and Koulish, J. 
Forthcoming. Data Use for Continuous Quality Improvement: What the Head Start Field Can 
Learn From Other Disciplines, A Literature Review and Conceptual Framework. OPRE Report 
# 2014-77. Washington, DC: US Department of Health and Human Services, Administration 
for Children and Families, Office of Planning, Research and Evaluation. 

Each organization may begin the systematic process at a different place in 
the cycle. For example, organizations receiving government funding typically have a 
considerable amount of data already collected (through the enrollment process, for 
example). They may begin their systematic process with the step "Analyze Data." 
Once an organization begins the cycle, there is no longer a starting place because 
the loop is continuous. The "Feedback” label in the center of the cycle shows arrows 
going back and forth throughout the cycle. This is to emphasis the importance of 
communication and engagement of staff in the process. The cycle emphasizes the 
need to engage staff and stakeholders throughout the organization at each step to 
ensure they are paying attention, gathering information, considering the data, and 
contributing interpretations about the data (see Part IV to get started on involving 
your staff, engaging your board and Policy Council, and encouraging parents). 

Organizations should not use the process as though each stage of the cycle is 
happening at a particular time of the year, but rather as though all steps of the 
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cycle are happening all the time. In Head Start programs, staff are gathering data 
every day, each time a family services worker takes a new application, each time a 
home visitor writes notes about new skills a mom demonstrated, each time a 
teacher makes observation notes about a child sharing, and each time attendance 
is recorded. The cycle helps your program envision those routine data collection 
efforts as part of a bigger process for improving quality. 

Unless your organization intentionally and regularly examines the data you 
already collect, then you might not already be analyzing, reviewing, or synthesizing 
it on a regular basis. You may decide that "regular” is defined as weekly, biweekly, 
monthly, or quarterly, and it may depend on the kind of data. For example, you 
may want someone examining (the "analyzing data," and "reviewing and 
synthesizing" steps) attendance data at least weekly so that patterns of absence 
can be observed quickly, staff can determine an appropriate course of action (the 
"prioritize and plan" step), and staff can follow up with families immediately to see 
what supports can be offered to get the children back in the classroom or 
participating in play groups (the "implement plan" step). In this scenario, the four 
steps of the cycle described may happen within a single day or over a few days. 

Likewise, monitoring performance is a continuous process. When your 
organization set your objectives, you would make sure they are Specific, 
Measurable, Attainable, Realistic, and Timely (SMART). 21 This means you would 
have included the outcomes you hope to achieve, the ways that you intend to 
measure progress toward the outcomes, and the timeline for achieving that 
progress. When you engage in your internal monitoring of performance, you are 
checking regularly to see how much progress you have made and whether it is as 
much as planned for the given period. For example, if you just trained teachers on 
a curriculum element last week, you may expect that they are implementing 
elements from the training, but you may not expect them to be highly proficient 
yet. 


When you evaluate outcomes, you are likely to pin that evaluation to a 
particular amount of time: has your team made the progress you had hoped in the 
amount of time expected? You probably won't use the same amount of time for 
each type of outcome; you must decide the appropriate amount of time for 
measurable progress to occur. Outcomes may be evaluated against the yearly 
benchmarks, some against biannual benchmarks, etc. Remember that you are 
continuously monitoring to see how much progress you have made and if you need 
to adjust service delivery or take management actions, but the "evaluate 
outcomes" stage is the official determination of whether you met the benchmark in 
the allotted time. 


21 PMFO, Foundations for Excellence, 12. 
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Next, we talk about objectives you may have, outcomes you may be trying to 
reach, and decisions you may need to make. We provide three examples drawn 
from our research in Head Start programs that demonstrate the kinds of data you 
might use and how you might use it to improve quality in your program: early math 
skill improvement for four-year-olds, child-teacher interactions, and making hard 
choices when budgets decline. 

Learni ng How to Use the Conti nuous I improvement Cycle 
1. Early Math Skill I mprovement for Four-Year-Olds 




Revisit Goals and Objectives 




GOAL: 


Children will use math regularly and in everyday routines to count, compare, relate, 
identify patterns, and solve problems . 22 

SCHOOL- READINESS GOAL: 


Four- year-olds will continually improve their early math skills throughout the year. 




Gather Data 


Teachers are observing and noting children's skills from the first day of class. They 
use their curriculum-aligned data system to indicate when children demonstrate 
mastery of particular skills and how often they engage in activities supporting 
specific math concepts. They continuously track student’s progress to determine 
how to individualize teaching to help students to continually improve. 

Three times a year, the Ed Coordinator works with the person in the 
organization supporting data analysis to gather aggregated data by classroom and 
by site to determine the children's progress overall. 


22 See the "Cognition and General Knowledge" goal examples provided in National Center on 
Quality Teaching and Learning. "School Readiness Goals for Preschool Children in Head Start 
Programs: Examples from the National Center on Quality Teaching and Learning." US 
Department of Health and Human Services, Administration for Children and Families, Office 
of Head Start, n.d. 
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ifi 


Analyze Data 

V 


An Outcome That Needs Attention 


Each time your program gathers the aggregated data, staff examine the data to 
understand the children's assessment scores. After the second data collection, your 
team discovers that four-year-olds are not improving their math skills at the rate 
that you had hoped. 

Child Groups That Need Attention 


When you examine the data by gender, you discover that boys are generally 
improving in their math skills more slowly than girls. 

Now you must figure out what factors might be contributing to these 
differences and what you can do about it (possible improvements for service 
delivery or technical assistance and training for staff). 


Review and Synthesize Data 


Child Groups That Need Attention 

You could start by asking yourself and your team "What might be some of the 
reasons that four-year-old boys would perform differently in math than four-year- 
old girls? Are there differences in the children other than their genders? Are there 
reasons the children might experience your program differently?” 


After taking a closer look at the data, you might discover that more of the 
boys than the girls also attended your program when they were three years old. 
They started the year doing better in math than the girls. They haven't improved as 
much, but their development level is about the same. You may be relieved that the 
girls and boys are actually performing about the same, but now you worry that the 
children who started in your program when they were three years old are not being 
challenged enough. 

Identification of Promising Practices 

You and your Ed Coordinator look at the data by teacher to see if some teachers 
are having more success than others in helping these more advanced children to 
progress. Your Ed Coordinator discovers that one of your experienced teachers does 
appear to have more success in advancing the four-year-olds' development. The Ed 
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Coordinator approaches this teacher to determine if she has a particular strategy 
and it might be different than other teachers' strategies. The Ed Coordinator finds 
out that the teacher had identified children who have been in the program longer as 
needing to be challenged more, and she has been individualizing her teaching to 
accomplish that (a promising practice). The teacher has also been supplementing 
the curriculum materials because she found they were not challenging enough for 
these children (another promising practice). 

Identification of Improvements for Service Delivery or Training Needs 

The Ed Coordinator checks in with the other teachers to better understand the 
differences between the classrooms where the children are advancing more and 
those where the children are advancing less. 


Prioritize and Plan Actions 


As a result of this data analysis, review, and synthesis, the Ed Coordinator makes 
four recommendations to the Director: 


1. In the next all-teacher meeting, the teacher with curriculum supplements 
should present on his or her materials and techniques to help spread the 
promising practice. 

2. Two teachers need additional training on individualization for children. 

3. In future years, all teachers should be made aware of the children in their 
class who attended Head Start the previous year (change in service 
procedures). 

4. The Curriculum Team should investigate if curriculum supplements are 
available for these children (change in service delivery). 


The Head Start Director will need to determine which of these actions can be 
taken right away and which ones will need to wait based on the availability of 
funds, need for approval from the Policy Council or the governing body, or other 
operational issues. When the Director approves the actions, target dates for 
completion and benchmarks for improvement should be set. 


Implement Plan 


Based on the decisions of the management and leadership, staff will take the 
actions necessary to implement the approved plans. Assuming that all four actions 
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have been approved, the teacher with the promising practices will present at the 
next staff meeting and the Ed Coordinator will help the two teachers access 
approved individualization training. The Ed Coordinator and the Curriculum Team 
will (1) begin planning to assure that next year teachers know which children are 
continuing in the program and (2) begin investigating alternate math curriculum 
options. 


Monitor Performance 


Spreading the promising practice and obtaining additional individualization training 
were supposed to happen in the short term. According to the plan, both items 
should have occurred two months after implementation began. Monitoring 
performance means checking to see (1) did the training happen and (2) did the 
training have the intended effect, that is, ensuring that other teachers are using the 
promising practices correctly and the weaker teachers have improved their child 
individualization. Monitoring for the intended effect would have to wait until an 
effect could be expected. For example, teachers will need some time to practice the 
new skills they have learned and incorporate them into their routines before any 
progress can be expected. 


The longer-term action items won't be fully implemented until the next year, 
but it will be important to check that progress is being made. Has the Curriculum 
Team met? Has someone begun working on a process to ensure teachers know 
about their children's participation status? 


ifS 

Evaluate 

l! 




V 


Were you able to improve four-year-olds' math skill development throughout the 
school year? You will have to wait until next year to determine if the outcome has 
changed. 



Do you want to keep your math skill development school-readiness goal the same 
or change it? Are there any parts of your goal that need to change based on 
external rules or policy changes? Do the changes you have made internally require 
a change in your goals or how you measure them? 
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2. Improving Child-Teacher Interactions 

The Classroom Assessment Scoring System (CLASS) is an observational 
assessment tool used during Head Start monitoring assessments to determine the 
quality of interactions between teachers and children across three domains: 
emotional support, classroom organization, and instructional support . 23 The CLASS 
scores are used as part of the Head Start Designation Renewal System as one 
criterion to determine if your program is higher or lower quality. Thus, obtaining 
higher CLASS scores could be seen as an effort to comply with the quality 
requirement. However, the CLASS scores could also be considered as an 
opportunity to improve classroom interactions; the staff of many programs that we 
spoke with held this view. 


In the example below, your program has not previously had objectives 
around CLASS scores, so you are going to begin with the "Gather Data" stage in the 
continuous improvement cycle. 


Gather Data 


You contracted with a few consultants who were certified as reliable in CLASS 
assessment to rate each of six preschool classrooms across three sites where your 
program operates. You now have the scores for each of the three domains for those 
six preschool classrooms. Your consultant provided you the scores in a spreadsheet. 
Figure D shows an example of the data you have. In contrast to Figure D, the 
spreadsheet provided by the consultant would have each subdimension with the 
domains. (The numbers used in Figure D do not represent any particular program. 
They are to demonstrate principles only.) 


23 National Center on Quality Teaching and Learning. I mproving Teacher-Child Interactions: 
Using the CLASS ™ in Head Start Preschool Programs. Washington, DC: US Department of 
Health and Human Services, Administration for Children and Families, Office of Head Start, 
2013. 
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Reading a Spreadsheet 


The numbers on the left are the row 
headings. The letters on the top are the 
column headings. When you are discussing 
numbers in a spreadsheet you usually refer 
to them by their column and row heading. 
For example, if you want to point out the 
highest Instructional Support CLASS score, 
you might say, "Look at E5, the Egret 
classroom scored 3.1." 


Figure D. Hypothetical CLASS Domain Scores from Preliminary Consultant 
Assessment 



A 

B 

c 

D 

E 

1 

Site Name 

Classroom 

Name 

Emotional Support 

Classroom 

Organization 

Instructional 

Support 

2 

Central 

City 

Robin 

5.5 

5.2 

2.1 

3 

Central 

City 

Bluebird 

5.2 

4.9 

2.5 

4 

Central 

City 

Sparrow 

5.1 

4.9 

2.3 

5 

Suburban 

Egret 

6.2 

5.8 

3.1 

6 

Suburban 

Pelican 

5.4 

5.1 

2.3 

7 

Rural 

Swan 

5.8 

5.6 

2.7 


Analyze Data 

IS 


St 


v 


You ask the staff member who typically handles your data to create some graphs to 
help you compare the data in different ways. You want to be able to better see the 
range of the scores by site (Figure D-l), and the range of the scores by domain 
(Figure D-2). 
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You note that the classroom in the rural site tends to have relatively high 
scores (a relief because that teacher and assistant are alone at that site). You also 
note that the central city classrooms are scoring the lowest. 

Looking at the domain scores by classroom (Figure D-l), it is easier to see 
that Egret has better scores on all domains than the other classrooms. 

Looking at the classroom scores by domain (Figure D-2), it is easier to see 
that all classrooms score the best in emotional support. 

But you are still not sure what this analysis means. Why are some 
classrooms performing better than others? Why are all classrooms performing 
better on certain domains than others? You put discussion of these data and these 
questions on the agenda for the next regularly scheduled data review meeting (See 
Part V for description of a data review meeting). 

Figure D-l: CLASS Domain Scores by Classroom Site 


7 



Robin Bluebird Sparrow Egret Pelican Swan 

■ Emotional Support ■ Classroom Organization ■ I nstructional Support 


36 


Figure D-2: CLASS Domain Scores by Domain 



(2 

Review and Synthesize Data 


Sv 


V 


At your next data review meeting, the Ed Coordinator presents Figures D-l and D-2 
to the review team. He or she also presents Figure D-3, which shows data from 
Office of Head Start showing grantee CLASS scores of Head Start programs from 
the most recent monitoring group. Finally, the Ed Coordinator says that she has 
spoken with each of the teaching teams and the two site directors (at the rural site, 
the teacher is also the site director) to get their perspectives on the CLASS 
observation notes and descriptions of child-teacher interactions in their classrooms. 


The Ed Coordinator provides her perspective on the score levels and possible 
reasons for differences. She reminds everyone that CLASS scores are assessed on a 
seven-point scale. She notes that the pattern among classrooms to score best in 
the emotional support domain and worst in the instructional support domain is 
reflective of the national picture shown in Figure D-3. Regarding the differences 
across classrooms, she explains that the teacher in the Egret classroom has 
participated in more professional development around CLASS than her peers; the 
teacher in the Sparrow classroom is new to Head Start and hasn't had any formal 
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training on the CLASS. The program had to hire her after the pre-service training 
days because of turnover in the classroom. 

The Ed Coordinator asks if anyone else from the team can think of other 
possible reasons for differences. The Director asks if classrooms with higher CLASS 
scores also have children improving more in their developmental domains. The Ed 
Coordinator responds that she has not looked at that yet. The Director requests 
that she bring that information back to the next meeting before the group works on 
prioritizing or planning any actions. 

The Ed Coordinator used guidance provided by the National Center on Quality 
Teaching and Learning to guide the discussion about the CLASS scores with the 
various teams within the program. See "Understanding and Using CLASS for 
Program Improvement ." 24 

Figure D-3: Office of Head Start CLASS Descriptive Statistics 
Distribution, by Domain (2013) 


National Distribution of Grantee-Level Domain Scores 

Domain 

Lowest 10% 

Median (50%) 

Highest 10% 

Emotional Support 

5.5417 

5.99 

6.44 

Classroom Organization 

5.0556 

5.62 

6.17 

Instructional Support 

2.1061 

2.71 

3.35 


Source: Office of Head Start, n.d. A National Overview of Grantee CLASS Scores in 2013. 
Washington, DC: US Department of Health and Human Services, Administration for Children 
and Families, Office of Head Start, http://eclkc.ohs.acf.hhs.aov/hslc/data/ class- 
reports/ docs/ national- class- 2013-data.pdf . 

This Head Start program holds monthly 25 data review meetings where all the 
department coordinators come together to share the most relevant, timely data 
they have about activities in their areas. Next month, the Ed Coordinator will 


24 National Center on Quality Teaching and Learning. Understanding and Using Class for 
Program Improvement. Washington, DC: US Department of Health and Human Services, 
Administration for Children and Families, Office of Head Start, 2012. 
https://eclkc.ohs.acf.hhs.aov/hslc/tta-svstem/teachina/docs/class-brief.pdf . 

25 The expectation is that coordinators will be monitoring the data about services in their 
areas of responsibility on a daily basis. The organization has found that monthly reviews at 
the coordinator level work best for discussing routine programmatic objectives. However, if 
a coordinator were to note an issue that needed to be addressed more quickly, he or she 
would be expected to notify their supervisor immediately. 
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provide information to the group that compares the CLASS scores and the children's 
developmental progression. After that discussion, the group will either decide that 
they need more information or that it is time for the Ed Coordinator to meet with 
the site directors to develop a suggested strategy for improvement so that the 
organization can prioritize and plan actions. And the cycle will continue. 

Cutting budgets is always hard, but when you have data, budget cutting can 

become an opportunity to realign your strategic direction. 

3. Prioritizing Allocations When Budgets Decline 

Head Start Directors must make decisions every year about what to prioritize in 
their budgets. Allocating across many important needs is always challenging, but 
choosing what to cut when budgets decline can be even harder. Having data about 
your community, your needs, your successes, and service usage helps to make 
strategic budget decisions. 

Recently, Head Start programs nationwide were faced with making hard 
choices about what to cut due to budget reductions at the national level. Some of 
them used their data to cut parts of their program that were not performing well, 
were being under-utilized, or did not align sufficiently with needs. These Directors 
said that though budget cuts were hard, the need to cut also gave them an 
opportunity to use their data to make strategic decisions with minimal resistance. 
We discuss two of the decisions to make changes in service delivery here. 

When Head Start programs are provided with budgets that don't meet their 
planned strategies, they must decide how to cut programs. Two basic strategies 
exist: cut across all parts of the program or strategically cut some parts of the 
program. Head Start programs generally use both options. Distributing cuts across 
all service delivery areas means that everyone bears some of the pain. One method 
to cut across all areas is to reduce the number of days of service provided. This 
leads to fewer days of learning for children, fewer days of support for families, and 
fewer days of work for staff. Cutting some parts of the program strategically means 
that not everyone shares the pain evenly, but it provides the opportunity for what 
is left to operate more efficiently and effectively, and perhaps to better meet the 
community needs. Head Start programs must decide what works best based on 
their own community and organizational needs at the time that cuts must be made, 
but it is worth considering all the options. 

Strategic cutting is only an option if you have the data to make strategic 
decisions. Strategic decisions require that data be available in units smaller than 
the whole. How does your program align with needs in particular communities, for 
particular kinds of families, and for particular kinds of children (See discussion 
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about disaggregation of data in Part IV)? How uniform is quality across your 
program? Do you have parts of your program that are not performing as desired? 
Do you have sites that are consistently not meeting the quality objectives you have 
set? Your overall quality is affected by the quality of each site where your services 
are provided. You are concerned that children are not getting the quality of care 
you strive to meet. Knowing the quality at each site allows for strategic cutting of a 
site (Site D in the example graphic) if it is consistently failing to meet standards. 


Site A Quality 


Site B Quality 


Overall Quality 


Site C Quality 


Site L Quality 


Eliminating Some Slots 

Head Start programs have many types of service delivery models, and in many 
cases they use mixed approaches in the same program. They may hire their own 
staff and run programs independently; they may pay for classrooms in community- 
based child care or pre-K programs but provide their own staff; they may contract 
with community- based programs to provide slots for Head Start children who are 
mixed in with other children, in which case the teachers are staff of the community- 
based organization; they may contract with family child care providers to offer 
Head Start slots; and they may provide the slots needed in other ways. Head Start 
programs are creative in their partnering to provide slots for children wherever is 
best in each community. All of the slots need to meet the Head Start standards. 
Quality control can be more difficult in some slots than others. 

In this example, one Head Start program was using the mixed approach of 
providing some preschool slots through their own teachers at sites around the 
community and providing some preschool slots contracted through community- 
based child care programs where the staff were employed by the community- based 
program. This Head Start program had been collecting data for a few years on the 
developmental improvement of the children and the classrooms' CLASS scores; it 
had also been keeping tabs on the centers' state licensing ratings. Program staff 
reviewed data on all the children the program served and on all the sites providing 
the service every three months. 

They observed the following when comparing across the sites: 
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> Quality across sites was inconsistent in the level of state licensing 
maintained. This inequity was increased when the state raised expectations 
about quality standards. 

> Quality across sites was inconsistent in CLASS scores that programs were 
achieving, and the lower-scoring sites were not improving. 

> Data were not being reported consistently from all sites, making it difficult to 
compare quality and monitor child outcomes. 

When the Head Start management staff examined the inconsistencies across 
sites more deeply, they found that the sites where staff other than their own were 
providing the services were the ones with the lowest indicators of quality. However, 
the teachers met the same types of credentials, had access to the same training, 
and in theory had the same supports. 

Digging deeper, the Head Start administrative staff discovered that even 
though they were providing the community-based sites with funds to provide 
planning and data-entry time for the teachers, the funds were being prioritized 
differently in the sites and the teachers were not actually receiving that time. 
Despite attempts to help the community-based sites understand the importance of 
this time and restructure the teacher day to accommodate planning (such as by 
hiring floaters that could cover classrooms for a few hours, allowing teachers to 
plan during the day instead of after it), the community-based programs seemed 
unable to structure their day in a compatible way. Though the Head Start 
administrators could not say that this lack of planning and data-entry time was the 
definitive factor in the quality differences, differences' persistence and the similarity 
across teachers and other supports pointed to that as the primary problem. 

The Head Start administrators had become increasingly wary of maintaining 
the relationship with the community-based sites. First, the Head Start program was 
committed to providing a consistent level of quality across the community, and 
their data showed that this was not happening. Second, they knew that lower levels 
of quality in any of their sites could lead to repercussions for their whole program. 
But it is hard to end relationships once you start them, and the partnerships were in 
place to meet specific community needs. 

In this case, the need to make choices about where to cut provided an 
opportunity. The Head Start administrators already had the data had identified 
weaknesses, and knew where strategic cuts could be made to benefit the most 
children and families. After consulting with the Office of Head Start about their 
options, they presented the idea to the Policy Council and governing body to 
remove the weakest programs to balance the budget. 
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As funds were restored or secured through new sources, they had data about 
what worked and what didn't. They used that data to find new partners and develop 
new strategies to provide the highest-quality services to the children in their 
community. 

Shifting from Home-Based to Center-Based Care 

Sometimes Head Start and Early Head Start early learning services are provided in 
a center- based environment. Sometimes programs focus primarily on parents as 
the child's first teacher and provide home visitors to regularly spend time with 
parents and children, with the goal of teaching parents how to support their own 
children's development, form attachments, and interact positively with their 
children. In both models, the child receives developmental engagement and support 
and parents learn how to better support their children. 

In this example, one Head Start program developed a mixed approach, but 
by examining their data over time, found that community needs had changed. 
When the need to cut budgets arose, they took that opportunity to make choices 
about their service delivery options, particularly for the Early Head Start program. 

Over the past few years, the Head Start program had been increasing the 
numbers of Early Head Start slots available in the community. The program had 
been focusing primarily on the home-based option because early research indicated 
that parents in the community wanted to stay home with their children in the early 
years. This Head Start program examines data from many sources across its 
organization on a monthly basis. Staff had noticed that over the past two years, 
most families using the home-based program were on the waiting list for the 
center-based program. As soon as a center-based slot became available, families 
were switching from the home- based program to the center- based program. Staff 
inferred that the home-based program was a second choice option for families; they 
knew that by being affiliated with the program already, they could more easily get 
into a center- based slot. 

When we created the home-based option, more parents in our 
communities wanted to be at home with their children. Now, they need or 
want to be at work, so center-based options align more closely with 
community needs. 

The Head Start program realized that this was not a productive approach to 
providing services. Families in the home- based program were not really invested in 
the home-based principles. They wanted their children to be in a center-based 
program. The frequent switching from home- based to center- based programs was 
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also hurting the program's ability to effectively set goals with parents and help the 
children and families. 

This Head Start program's ability to track its data across its program options 
helped the administrators understand that community needs had changed. They 
decided that the most strategic budget cut was to reduce the home-based option. 

When budgets increase again, staff will need to reflect on their current data 
to recommend the most strategic options for adding back slots or changing service 
delivery to reflect the current needs of their community. 


HIGHLIGHTED RESOURCES 
Technical Assistance Planning Series in Head Start 

"Planning in Head Start." Office of Head Start. 

The Head Start National Technical Assistance Center for Performance Management 
and Financial Operations developed these five papers to support programs in 
developing and implementing a planning system. These papers focus on goals, 
objectives, outcomes, progress and action plans and the relationship between 
them, school- readiness goals, family and community engagement, and program 
examples. 


http://eclkc.ohs.acf.hhs.aov/hslc/tta-svstem/operations/plannina.html 


Understanding and Using CLASS for Program I improvement 

National Center on Quality Teaching and Learning. 2012. Understanding and Using Class for 
Program Improvement. Washington, DC: US Department of Health and Human Services, 
Administration for Children and Families, Office of Head Start. 
https://eclkc.ohs.acf.hhs.aov/hslc/tta-svstem/teachina/docs/class-brief.pdf 

The CLASS is an observational assessment tool focused on teacher-child 
interactions. The CLASS is used by the Office of Head Start as one measure of 
program quality, but it has other uses as well. The Head Start National Center on 
Quality Teaching and Learning developed this brief to help Head Start programs 
understand what CLASS results mean, the different ways CLASS results can be 
used, and how CLASS results can be reported and shared. 
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PART I V 


Prepari ng Your Data For Use 


Of course, understanding what should be done is much different than figuring out 
how to do it. This part of the resource guide provides guidance on making sure that 
your staff and data are capable of exploring deeper meanings. We briefly provide 
tips on ensuring your data quality in the dimensions of accuracy, completeness, 
validity, and reliability. We also suggest dimensions by which you may want to 
disaggregate 26 your data and thoughts on understanding your data. (See Part V for 
additional information about engaging staff and stakeholders in use of data.) 

Ensuring Data Quality 

Data quality refers to the accuracy, completeness, validity, and reliability of your 
data. Assuring the quality of your data requires that staff at all stages of the data 
collection, entry, and analysis process have sufficient understanding of the data 
instruments, data, and the data systems. Training and procedures are important 
supports for assuring quality (more discussion on this can be found in Part V). 

Validity 

By definition, "a measure is valid to the extent that it measures what it is intended 
to measure ." 27 In practice, there can be many ways to capture information on a 
particular item. For example, there are different questions that can assess 
individual motivation or interest in learning, strength of family or teacher 
relationships, perceptions of personal abilities, or areas of improvement. For these 
reasons, it is important to use data collection instruments, scales, and other tools 
that have been tested, particularly with specific populations of interest (such as 
different ages, racial/ ethnic groups, etc.). 

Data quality begins with the validity of the data collected. Data validity 
comes from selecting the right way to answer your questions of interest, including 
questions about what kinds of instruments you use. Should you use an instrument 


26 Data may be aggregated or disaggregated. For example, individual child-level data would 
be the lowest level of disaggregation (the smallest level); data about all the children in your 
entire program would be the highest level of aggregation for your program. In between, 
there are many ways to group your data to learn more about the children you are serving. 

27 Rossi, Peter H., and Howard E. Freeman. Evaluation: A Systematic Approach - 5 th Ed. 
Sage Publications, 1993. 
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that is norm- referenced or criterion- referenced? The answer to that question and 
others requires consideration of many factors beyond the scope of this guide. One 
resource you may find useful is Early Childhood Assessment: Why. What, and How 
from the National Academies Press . 28 

For example, you don't want to measure children's developmental progress 
using an instrument designed to measure teacher interactions with children. That 
is, if you want to know how much a given child has progressed in literacy skills 
during the year, you do not review his or her improvement in CLASS scores. 

Reliability 

By definition, a measure or value "is reliable to the extent that, in a given situation, 
it produces the same results repeatedly." But no measurement instrument is 
perfectly reliable. For example, measurements of weight and height, produced with 
instruments such as scales similarly calibrated, are likely to produce more-reliable 
results than measures of attitude or personal beliefs . 29 

Reliability refers to both the measure itself and the individuals doing the 
measuring. For example, a scale may consistently measure weight, but if the 
individual being weighed sometimes wears shoes and sometimes does not, then the 
measure of weight will vary. If multiple persons are overseeing the weighing, then 
they all need to be trained on the same procedures— either shoes on or shoes off— 
to ensure the reliability of the weight. 

Similarly, you need to ensure the reliability of the data collected by your 
program by ensuring that individuals collecting all kinds of data have been trained 
to follow the same procedures. You don't want some of your data to be different 
because one staff member was asking slightly different questions or recording 
observations differently than another person. You need to be sure that all data 
collected for the same purpose mean the same thing. 

Accuracy and Completeness 

Larger programs or those with more computerized your systems have more staff 
likely to have data entry included in their duties. Programs with more staff entering 
data need more procedures to assure that the data are accurate and complete. 
Staff need to understand the expectations for how often data must be entered and 
key time periods when all data must be in the system to assure that analyses can 
be performed on complete data. If managers are reviewing data daily, they will be 
interpreting the data incorrectly if only half of staff are entering their data daily. 


28 Snow, Catherine E., and Susan B. Van Hemel. Early Childhood Assessment: Why. What, 
and How. Washington, DC: National Academies Press. 2008. . 

http:// www.nap.edu/openbook.php7record_ id =12446 . 

29 Rossi, Peter H., and Howard E. Freeman. Evaluation: A Systematic Approach. 
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Programs with more data systems need to provide more instruction to staff 
about procedures for data extraction from each system (because systems may be 
different). Every system has its quirks; sometimes these are well-documented in 
instructions and sometimes they are not. For example, if you want to disaggregate 
your data, you need to know how to do it correctly in the system you are using. 
Frequently, you need to request data in a series of stages or commands. The order 
of those stages or commands usually influences the analysis results. If you use the 
same stages or commands in a different order, you get different results. However, 
when you review the results, you might not be able to tell that the data are 
incorrect. 

If your results vary greatly from month to month or quarter to quarter, you 
may want to revisit your procedures. Issues may also occur when aggregating your 
data at a higher level than usual. For example, if you normally examine your data 
by funding stream (children supported by Head Start separated from children 
supported by state pre-K funds), then you may experience duplication of 
information when you combine the data (at least one Head Start program we spoke 
to cited this as something to watch for while extracting data from systems typically 
used by Head Start programs). If children are supported by multiple types of 
funding, their data may need to be entered in the system more than once. 
Accordingly, data on all children may include some children multiple times. 

We provide the following tip sheet for you to share with staff to highlight and 
remind them of important elements of data quality. The tip sheet is most 
appropriate for staff overseeing data entry, staff training others on data collection, 
or staff responsible for data extraction or interpretation. 
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Elements of Data Quality Tip Sheet 


Considering Your 
Data Quality: 
Accuracy 

Data Entry 

Individuals entering data or 
capturing data through notes need 
to understand where to enter what 
pieces of information. Data 
systems should control accuracy 
by limiting the number of 
characters that can be entered, 
allowing only numbers or words, 
or setting allowable ranges for 
information. Data samples should 
be checked regularly and everyone 
should receive training on the data 
systems they are using. 

Data Extraction 

Accurately entered data may 
become inaccurately represented 
when it is extracted from its 
system. Duplicate children or 
families may appear because 
multiple funding sources are being 
tracked. Data may be grouped 
incorrectly if the user does not 
have sufficient training. 


Considering Your 
Data Quality: 
Completeness 

Data should be checked for 
completeness before analyses are 
run. Does the data exist for every 
child, every classroom, every 
home visit, and so on? Has data 
been entered for the whole period 
under consideration? Has data 
been entered for every period 
under examination if you are 
comparing across time periods? 


Considering Your 
Data Quality: 
Validity 

Not everything you want to 
measure will have a tested 
instrument. This is acceptable, but 
you still need to be sure your data 
measures what you thought it 
would. 

Are you drawing the right kinds of 
conclusions from your data? 


Considering Your Data Quality: Reliability 

Are staff who are gathering the data sufficiently trained to gather it the same way 
every time? Are all staff gathering and documenting the data the same way? For 
example, some data are collected through observations. Would different staff 
members make the same notes or rate the activity the same way based on what 
they observe? 
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Disaggregating Your Data 


Head Start programs operate in a diverse set of communities, implement many 
models, and serve a wide variety of children and families. Consequently, staff may 
find it useful to break out data on programs, staff, children, and families by 
characteristics of interest. This process, called data disaggregation, can highlight 
differences between groups in Head Start experiences and areas in need of 
attention. The lowest level of data disaggregation is at the individual child or family 
level. Data at that level are certainly important, but the middle levels of data 
disaggregation can be most helpful in identifying patterns or differences in service 
delivery and outcomes. These levels of disaggregation may suggest questions for 
future exploration and illuminate the type of data needed to further understand 
differences between groups. 


Table C. Possible Client and Service Characteristics to Use in Disaggregating Data 


Child Characteristics 

Gender 

Age 

Race or ethnicity 
Disability status 
Health insurance status 


Staff Characteristics 

Type or role 
Qualifications 

Participation in professional development 
Race or ethnicity 
Direct or contracted 


Family Characteristics 

I ncome 

Highest level of education 
Language spoken at home 
Head Start volunteer status 


Classroom Characteristics 

Age composition 
Child to staff ratio 
Facilities 

Presence of volunteers or aides 


Site Characteristics 

Direct or contracted 
Partner or delegate 
Neighborhood served 
Total enrollment and staff size 


Management and leadership staff may disaggregate data using several 
characteristics to guide quality improvement efforts. For example, returning to the 
IEP/IFSP example used in Part I, you may want to examine whether children are 
receiving disability services, looking at such site characteristics as whether the site 
is one you operate directly or you contract, whether the site is a partner or a 
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delegate, and where the neighborhood where the site is located. You might create a 
table similar to Table B in Part I for each subgroup of interest. If you discover that 
sites you contract with, partner with, or delegate to have different rates of access 
to services than sites you operate directly, then you would consider why that is the 
case. Do they need more guidance? Are the sites located in places where it is 
harder for parents to access services? Do the sites have a higher proportion of 
parents that do not fluently speak English and therefore may have a harder time 
accessing services? These questions point to the importance of careful 
interpretation, which we turn to next. 

Analysis and Interpretation 

Continuing the example of the lEPs and IFSPs, we briefly discuss the iterative 
process of understanding data. Once you know that your data has been collected 
through the appropriate tools, that the individuals collecting it have done so in a 
reliable manner, and that the data have been entered completely and accurately, 
you are ready to begin analyzing and interpreting the data. Analysis is the technical 
portion of the process and includes extracting the data from the systems, linking 
the data together appropriately, and generating information. Some typical 
analytical techniques include determining how many of each item of interest you 
have (frequencies), the range of the items (distribution), the typical amount of each 
item (statistics such as mean and median), and associations between items (e.g., 
identifying that when one thing occurs, another thing usually occurs too). It is 
important that staff who run these analyses have a basic understanding of 
statistics. Computer programs make it possible for people who do not know what 
they are doing to produce numbers, but those numbers could be meaningless or 
cause incorrect interpretations. 

Sometimes the same staff who provide or manage the services know how to 
perform analyses, but sometimes they do not. They may not have received such 
training as part of their educational program; if they did receive it, they may not 
feel comfortable in their skills. However, the staff who provide or manage the 
services are best- positioned to interpret the data. Interpretation requires an 
understanding of how the services are delivered, the types of elements that may 
alter the amount of services delivered, the quality or timeliness of services, and 
reasons that services may not have been delivered as expected. 

The iterative process involves deciding which initial levels or groups of data 
you want (levels of aggregation or disaggregation) and the types of questions you 
want to answer. Then someone with an understanding of how to conduct analyses 
extracts the data, links data across systems, and produces statistics for review. A 
person or team of people familiar with how the service is provided review and 
interpret the data. The review of the data may lead to more questions, and data 
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reviewers send those questions to the person with expertise in data analysis. The 
analyst performs more analysis and sends the results to the review team, and the 
cycle continues as necessary. Sometimes the data needed for answering the deeper 
questions are already being collected, but sometimes more data collection is 
required. The following tip box, discussing the iterative process for using data to 
improve program quality, can be shared with staff. 


Using Data to I mprove Quality 

1. Understanding your data is an iterative process. Sometimes you have to 
keep digging to find the real issues. 

2. Analysis is not the same as interpretation. Analysis helps you determine 
what you need to be examining, but it doesn't tell you how or why. 

3. Interpretation requires an understanding of the processes, procedures, and 
actions involved in the service. If you are not delivering the service 
yourself, you should discuss with the people who are. 

4. Deciding what to do about an issue requires a full understanding the issue, 
the program requirements in relation to the issue, and the resources you 
have to address the issue. 


PMFO has produced an online, interactive guide on digging into data. It 
provides additional resources for reflecting on your data quality and completeness 
and considering the types of questions you might ask to understand your data 
differently. 30 


30 "Data on Head Start and Early Head Start: Digging into Data." Office of Head Start, last 
modified August 7, 2014, https://eclkc.ohs.acf.hhs.gov/hslc/tta- 
system/operations/data/data2. 
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HIGHLIGHTED RESOURCE 


Analyzing Outcome Information: Getting the Most From Data 

Hatry, Harry P, Jake Cowan, and Michael Hendricks. 2004. Analyzing Outcome Information: 
Getting the Most from Data. Series on Outcome Management for Nonprofit Organizations. 
Washington, DC: Urban Institute, http://www.urban.org/publications/310973.html 

This guide suggests ways to analyze outcome data to improve services and 
outcomes for individuals being served by a nonprofit or community program. It 
offers many strategies for disaggregating data and setting up useful comparisons to 
help a program learn for which groups or under what circumstances outcomes may 
be exceeding or failing expectations. The guide is written for users with basic or 
intermediate knowledge of math or simple statistics. 


51 


PART V: Engagi ng staff and stakeholders 

Valuing and I ncluding Many Perspectives 


OVERVI EW 

Research suggests that data use is most effective when organizations involve staff 
from all across the organization, engage their governing bodies, and encourage 
stakeholder involvement (stakeholders include those served by the organizations, 
such as families). In this section, we draw from both ideas in the research and 
examples of approaches given by Head Start programs to develop suggestions on 
how to get started(for example, including more individuals in data use efforts or 
using a variety of graphics to tell your story). 

Involving staff, engaging governing body and Policy Council members, and 
encouraging parents to become involved in data use can help organizations 
understand their data in new ways and enhance continuous improvement efforts. 
Individuals in these groups are likely to be less familiar with data; thus, they need 
training on how to understand the data, presentations of data that facilitate 
understanding, and data that are relevant to them. 

These descriptions are meant to provide your organization with initial ideas 
about how to include many perspectives in your organizational learning process. It 
isn't necessary to adopt all of these strategies, and how each strategy is 
implemented by different organizations will vary. As you think about how to adapt 
these strategies for your organization, it is important to keep some of their 
important characteristics in mind. First, all of these strategies show that your 
organization values data and the participation of others by devoting time to 
examining, discussing, and reflecting on data. Your organization will have to figure 
out when and how to carve out that time while the data are still relevant and 
timely. Remember, if you are not putting out fires all the time, you should have 
more time for making improvements. Teachers, however, need to spend a certain 
number of hours in the classroom. Those hours don't change just because they 
need to spend time outside the classroom engaging with information and other 
staff. Second, the success of these strategies depends on creating a safe, trusting 
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environment, one component of which is attention to staff concerns about data 
use . 31 

1. I nvolving Staff 

Here, we offer four strategies for involving staff at all levels in the continuous 
learning process. Individuals working at different levels and focusing on different 
parts of the organization bring different perspectives to an issue. Including staff 
from all across the organization enhances the organizational learning process. 
Remember, you want to learn from the routine: everyday activity and action 
provide an opportunity for learning, but your organization needs to create the 
pathways so that the individuals who are learning can contribute to the whole 
organization's learning. These strategies of data orientation, reflective supervision, 
data review meetings, and inclusion of data review in staff meetings provide those 
pathways. 

Data Orientation 

When individuals are hired into an organization, they often participate in a new- 
employee training to introduce organizational policies, standard operating 
procedures, and information about informal practices and program culture. It may 
also be helpful for organizations to update job descriptions in ways that clarify 
expectations around data use and analysis. Over time, this will reinforce the day-to- 
day importance of data as a core organizational value. 

Organizations that are data-informed typically use many procedures and 
strategies to introduce staff to data systems, data collection and reporting 
procedures, participant confidentiality principles, and ways to safeguard personal 
information. Ideally, the staff collecting and entering data will understand the 
purposes of the data, definitions of the particular data elements, and proper use of 
the data system. If the staff are expected to get data out of the data system, they 
also need training on how to appropriately extract the data and run reports. 


31 The "District and School Data Team Toolkit" referenced earlier in this guide, includes a 
number of hands-on tools and strategies for managing change and understanding concerns. 
See http://www.kl2.wa.us/CEDARS/Data/pubdocs/FullToolkit.pdf . See also "Early Learning 
Data Institute." Capital Region ESP 113. See also "Tips for Embracing Data" from PMFO that 
provides 10 key tips for creating a culture that embraces data, starting with a program 
vision that values data, supports staff professional development in data literacy, and 
effectively evaluates the data process. 

National Center on Program Management and Fiscal Operations. Tips for Embracing Data. 

US Department of Health and Human Services, Administration for Children and Families, 
Office of Head Start, n.d. 

http://eclkc.ohs.acf.hhs.gov/hslc/tta-svstem/operations/data/tip-sheet.pdf . 
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Common data extraction steps are needed 


to ensure consistency of data interpretation. 
Using different steps or putting steps in a 
different order to get the data can provide 
different results and data meanings. You may end 
up "comparing apples to oranges" across 
classrooms or sites without knowing it. 


Depending on the size of the organization, these strategies can take many 
forms. Some organizations may offer a formal workshop using supporting manuals 
or how-to guides with screenshots or sample reporting templates. In other 
organizations, such orientation may be more informal or ad hoc. Ideally, however, 
if your organization strives to be data- informed, you should have mechanisms in 
place to orient all employees around the use of data while understanding that the 
receptionist or food service provider may not need the same level of introduction as 
the Ed Coordinator or home visiting staff. 

Regardless of the approach your program takes to identify staff's training and 
professional development needs, the goal is to create repeatable behaviors and 
regular use of data in day-to-day discussions and decision-making throughout the 
organization. This will necessitate repeat or refresher workshops on data use and 
occasional check-ins with staff about what needs and supports are most helpful to 
successful engagement. 

Reflective Supervision 

The "culture shift" described in Part II is based on several new approaches, that is, 
ways in which organizations approach their work and relationships. Reflective 
supervision is a way of promoting a more supportive and collaborative relationship 
between supervisors and staff (as well as between staff and children or families). 
Reflective supervision is based on three basic principles: reflection, collaboration, 
and regularity. It is fundamentally different from the traditional supervisory 
relationship, which tends to focus on achieving specific standards, evaluating 
employee performance, and enforcing rules or policy. Though both types of 
supervision are necessary, this resource guide prioritizes the reflective, 
collaborative approach over the standards- based approach. 
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Reflective supervision is based on a foundation of honesty and trust. In this 
relationship, the supervisor acts more like a coach or teammate to help the staff 
member troubleshoot a problem, classroom situation, or interpersonal engagement. 
In these interactions, the supervisor is expected to exhibit active listening skills, 
ask thoughtful questions, and create a safe space for the conversation. Central to 
this relationship is suspended judgment of the situation or encounter under 
consideration. 

Reflective supervision, performed correctly, is an investment in staff 
development. Accordingly, it needs to occur on a regular basis, at a time 
convenient for both parties, and with a common agenda articulated in advance of 
the session. These sessions should occur at a time when distractions can be 
minimized; this enhances reflection and maximizes participants' ability to fully 
engage and support the process. 

Head Start staff across sites reported that they use variations on this 
reflective supervision approach to help teachers examine and reflect on information 
specific to the children in their classrooms and assessments of their interactions 
with the children (e.g., CLASS scores). Some Head Start programs have created a 
position in addition to the regular supervisor to lead these reflective supervision 
sessions. In some programs, the regular supervisor holds two different kinds of 
meetings with the teachers; every other week they meet for reflective supervision 
and every other week they meet to discuss performance and needed improvements 
from a more traditional perspective. The supervisors acknowledge that it is hard to 
balance the two roles, but think it is important to teachers' development and to 
productive data use to have these different kinds of meetings. 

Additional resources on reflective supervision can be found in Appendix D and 
at the Early Head Start National Resource Center. 32 

Data Review Meetings 

Finding a regular time for staff to sit down and review relevant program data is 
another important element of an organizational culture of continuous improvement. 
These sessions can use an approach similar to that of reflective supervision 
interactions and can occur one-on-one or in a group setting. The advantage of 
having these discussions in a group is that it maximizes the opportunity to hear 
different perspectives on the same information and develop a more vibrant peer-to- 
peer exchange. You might also consider different leaders for these sessions. For 
example, rather than pursing a typical top-down approach, you might have 
classroom teachers or family services coordinators facilitate the session because 


32 "Early Head Start National Resource Center." Office of Head Start, accessed November 5, 
2014, http://eclkc.ohs.acf.hhs.gov/hslc/tta-svstem/ehsnrc . 


55 


such an approach encourages ownership of the data. As with reflective supervision, 
it is important to be candid and open about what the data say and what individuals 
or groups of individuals can do to troubleshoot or solve problems in an area where 
improvement is needed. Each session should identify concrete action items and 
these should be revisited in subsequent data review meetings. The study group 
approach can provide a smaller, safer venue for exploring data. 

The CitiStat model offers a more formalized approach to data review that 
has taken root in many cities and state governments across the country. CitiStat is 
a data- informed model of government that emphasizes continuous review of data 
across government agencies. Typically, department leaders are convened every two 
weeks by the mayor or deputy mayor to review data that have been submitted 
before the meeting. Data are reviewed at each meeting and agency heads are 
expected to identify solutions and report on both progress implementing these 
solutions and any change in the data from the previous biweekly session. The US 
Department of Housing and Urban Development has a version of CitiStat called 
HUDStat. Under this program, the secretary of the US Department of Housing and 
Urban Development leads quarterly reviews "to drive improvement on priority 
goals" and to "prompt relentless focus on improving data quality and problem 
solving." 


HUDSTAT MEETI NG PRI NCI PLES 

> Understand that the data are not perfect but will evolve over time. They 
provide a basis for discussion. 

> Maintain environment of open and honest dialogue, even with imperfect 
results. 

> Practice "no surprises" with program offices and the support operations. 

> Limit "show and tell" and expand on problem solving using the collective 
wisdom and experience in the meeting. 

> Maintain a spirit of collaboration while not shying away from the issues. 

Source: Department of Housing and Urban Development. "HUDStat.” Performance.gov, accessed November 4, 

2014, http://archive-qoals.performance.gov/hudstat . 


The CitiStat and HUDStat models are readily adaptable to Head Start 
programs. These data review meetings can happen at many levels depending on 
the size of the program or organization. Certainly, in larger programs, the idea of 
departments heads coming together to review data and solve problems might be 
most akin to the CitiStat approach. However, this model (with little adaptation) 
could be applied in smaller programs or with different groups of people. For 
example, the Ed Coordinator could convene Head Start teachers quarterly to review 
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student progress (or perhaps once or twice a year, depending on the student 
assessment cycle). Alternatively, an interdisciplinary team could convene 
periodically to review data for a group of children who seem to be underperforming; 
this would bring together insights from classroom staff, home- based providers, and 
perhaps other supervisors, allowing the group to flag issues and propose a more 
integrated approach to accelerate the progress of children who may be lagging in a 
particular area. 

Data as Part of Regular Staff Meetings 

Incorporating data reports into regular staff meetings is an important means of 
keeping staff informed about the current state of progress of program priorities. 
Unlike the data review meetings described above, data use at staff meetings is 
likely focused on snapshots of indicators or trends you want to monitor fairly 
frequently, such as monthly attendance rates. Many Head Start programs have 
developed dashboards that summarize key data points in a single graphic. 

The "Assets and Opportunity Scorecard" 33 illustrates several reporting 
options and data visualizations on indicators related to assets and opportunity, 
including education. These formats could readily be adapted for Head Start 
programs. 

Additional information on scorecards, dashboards, and data visualization can 
be found in the "District and Data Team Toolkit - Component 3: Understand 
Issues". This component of the report includes an overview of data displays, 
attributes of a good data display, and a short section on dashboards. 34 

As you think about what kinds of data to report, it may be helpful to ask staff 
what kinds of reports are most useful to them and at what intervals. Providing 
reports based (at least partly) on data that staff are asked to collect will make 
these activities more meaningful to staff and engage them in thoughtful inquiry 
about the meaning behind the data. You might also consider having different staff 
present or report on data during staff meetings to create a greater sense of 
connection and ownership of the results. 

Experiment with different types of reports and formats (see Part III for some 
simple examples). You may wish to start with a small number of graphs or charts 
that are repeated at each staff meeting, but then supplement those as needed with 
data about areas where additional attention is required. The data review sessions 
provide an opportunity to go more in depth on some issues, but because they may 


33 "Assets and Opportunity Scorecard," Corporation for Enterprise Development, accessed 
December 12, 2014, http://assetsandopportunitv.org/scorecard/ . 

34 "Understand Issues," Capital Region ESD 113, accessed November 5, 2014, 
http://www.esdll3.org/Paae/1732 . 
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not involve all staff, it may be helpful to bring some of those issues to your regular 
staff meetings. 

2. Engaging your Policy Council, Governing Body, and Key Stakeholders 

Engaging your Policy Council, governing body, and other stakeholders is an 
important way to obtain additional or different perspectives on program operations 
and progress. Although Head Start requires some participation of, involvement of, 
and reporting to the Policy Council, governing body, and key stakeholders, required 
involvement may not tap into their full potential to contribute to greater program 
success. 

Routine Data Updates 

As with regular staff meetings, consider what kind of data are likely most 
interesting to different stakeholder groups. For example, Policy Councils and boards 
may have more of an interest than parent groups in data related to budgets, 
operations, and staffing. Data about child progress and performance are likely to be 
of interest to all groups. 

Important as well is to understand the level of data literacy for different 
groups. If your Policy Council includes parents with business experience, they are 
likely to be reasonably familiar with basic data presentations and may actually 
expect greater detail. Most likely, however, these groups will include a range of 
experience and familiarity with data, and you should experiment with different 
formats and presentations. You might consider creating simple handouts, such as a 
glossary of terms or an inventory of the kinds of data or indicators available. 

Head Start program governance training resources summarize the "Four A's" 
of reports: appealing, accessible, accurate, and audience specific. 
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The Four A's 

(from "Data in Head Start and Early Head Start: Digging into Data, Activity 6: 

Share Data") 

• Appealing: Keep it simple, clear, and visually attractive. Tell a story in a 
compelling way. 

• Accessible: Use an appropriate reading level and avoid jargon or acronyms 
that the audience doesn't understand. Use bullets rather than long 
narratives, and use the language(s) spoken by the audience members. 

• Accurate: Data must be free of error. The report must convey what the data 
actually say, not what you wish the data said. Proofread your documents to 
catch typos. 

• Audience-specific: Highlight the issues that the audience cares about. 
Consider the level of detail the audience needs and the audience's knowledge 
of the topic. 


Sources: National Center on Program Management and Fiscal Operations, n.d. Data in Head Start and Early Head 
start: Digging into Data. Washington, DC: US Department of Health and Human Services, Administration for 
Children and Families, Office of Head Start, http://eclkc.ohs.acf.hhs.gov/hslc/tta- 

svstem/operations/data/data2/ resources/ digging- into- data .pdf : National Center on Program Management and 
Fiscal Operations. “Data in Head Start and Early Head start: Digging into Data. Activity 6: Share Data." Last 
modified August 7, 2014. http://eclkc.ohs.acf.hhs.gov/hslc/tta-svstem/operations/data/data2/act6/activitv6.html 


It may also be helpful to look at reports from other programs to get new 
ideas about how to make presentations more intuitive and visually appealing. An 
inquiry to Head Start colleagues in other programs should get you started. Though 
Excel tables are an easy place to start, there are many new resources available for 
data visualization techniques, many free of charge. For an example, see Brian Suda 
and Sam Hampton Smith's post on Creative Blog. 35 See also the highlighted 
resource at the end of this section: a data-based report to a local community. 

Though much of this resource guide focuses on numerical data use, to 
engage stakeholders you might also incorporate photographs (assuming 
appropriate permissions have been secured) or short videos to enliven data charts. 

"Data walks" or "data walls " 36 are new techniques to engage community 
stakeholders in use and interpretation of data . 37 Although data walks, also known 

35 Suda, Brian, and Sam Hampton-Smith. "The 36 Best Tools for Data Visualization." 

Creative Blog, March 12, 2014. 

http://www.creativebloq.com/desian-tools/data-visualization-712402 . 

36 Slide 20 of this PMFO PowerPoint presentation provides a suggested set of instructions for 
using "data walls": National Center on Program Management and Fiscal Operations. 
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as data walkthroughs, have their origins as a technique for schools to improve 
instructional practice, the basic principle of posting different charts and data points 
on a wall or around a room is that engaging people in an interactive way is more 
likely to stimulate a conversation, spark questions, and contribute to group problem 
solving. 

3. Encouraging Families 
Routine Sharing of Child Data 

Engaging family members in their child's learning and development is essential for 
achieving the best outcomes for children. Head Start staff should view parents as 
partners and engage family members in meaningful ways. Family members can 
often provide valuable insights or other information to help explain or interpret 
classroom observations and results. The National Center on Parent, Family, and 
Community Engagement 38 defines the following three elements as necessary for 
sharing information with families: 

1. Families should have access to their child's individual learning and 
development information (e.g., by inviting family members to share 
observations about how their child learns, using several communication 
approaches to reach families, and providing data in a user-friendly format). 

2. Information that is shared should be understandable and meaningful to 
families and staff (e.g., by helping parents understand what the next stage of 
learning will be so that they can support development). 

3. Staff and parents should put the information into action (e.g., by sharing 
information about resources in the community that can help with a particular 
stage of development or suggesting day-to-day activities that can support 
particular development, such as matching pairs of socks or naming colors on 
street signs). 

Head Start programs traditionally offer many ways for families to get 
involved. Some of the programs we spoke with were using new technologies to help 


"Program Planning: Spotlight on Fiscal.” Office of Head Start, accessed November 5, 2014, 
http://www.wsaheadstarteceap.com/fileLibrarv/file 142.pdf . 

37 Skretta, John. 2007. Using Walk-Throughs to Gather Data for School Improvement. 
Reston, VA: National Association of Secondary School Principals. 
https://www.nassp.Org/portals/0/content/55499.pdf . 

38 National Center on Parent, Family and Community Engagement. Family Engagement and 
Ongoing Child Assessment. Washington, DC: US Department of Health and Human Services, 
Administration for Children and Families, Office of Head Start, 2011, 

https://eclkc.ohs.acf.hhs.aov/hslc/tta-svstem/familv/docs/familv-enaaaement-and-onaoina- 

child-assessment-081111.pdf . 
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families learn about their children. For example, some programs were purchasing 
electronic tablets that allowed teachers and home visitors to record videos of the 
children engaging in activities. In the home visiting setting, they could record the 
child and parent engaging together; in the center- based environment, they could 
record the child engaging with other children or accomplishing something on his or 
her own. In the home-visiting setting, the home visitor could play the video back 
immediately and point out to the parent the strengths of the interaction and what 
the parent should look for in the future. In the center-based environment, the 
teacher or family engagement coordinator could show the video to the parent 
during a home visit or parent conference to aid discussion of what the child is 
accomplishing or where the child is experiencing challenges. 

Staff in some programs could upload their videos into the child assessment 
data systems to document the developmental progress of the child. Some of these 
data systems allow parental access to parts of the system. At least one program 
found that parents were not using this feature much. Program staff were still 
investigating whether that was an effective avenue for providing information to 
parents. 

Another way to share information with parents is to use parent-teacher 
conferences to review progress and share observations about areas of strength and 
areas where improvement might be helpful. Equally important during these 
sessions is to listen to any concerns the parent or guardian might have. If teachers 
are sharing information about a formal assessment, it is important to present such 
information clearly and understandably. Many parents may not be familiar with 
scoring rubrics or other conventions that may accompany assessments. Thus, it 
may be necessary to translate into more concrete terms what an emergent skill set 
might look like. Compared to understanding technical developmental stages, it may 
be easier for parents to understand that their child can count to 10 but he or she 
often skips numbers between 11 and 20. Another way to help parents may be to 
explain the basic skills that define kindergarten readiness and where the child may 
need additional help, such as with basic writing or counting tasks; the ability to 
identify or replicate basic patterns among objects; the ability to match objects of 
various shapes, sizes, or colors; and the ability to follow simple safety instructions. 
Each Head Start program needs to work collaboratively with their local school 
systems to determine what these expectations are. 

Head Start programs may also need to be proactive in encouraging parents 
to sign up for and attend such meetings. Some may be reticent because language 
or cultural issues; others may be working multiple jobs and have limited 
availability. Head Start programs may need to be creative in seeking alternative 
ways to engage parents in their child’s learning and solicit their feedback, it is 
important that the program, through a combination of formal parent-teacher 
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sessions, informal opportunities for parents to come together more socially, and 
home visits, creates a welcoming and inviting atmosphere that supports parents in 
their role as a child's first teacher and creates a space that invites dialogue. 


HIGHLIGHTED RESOURCE 

DC Promise Neighborhoods I nitiative: Data Based Profile of the 

Kenilworth-Parkside Community 

DC Promise Neighborhood Initiative. 2014. I Heart Data: A Data-Based Profile of the 
Kenilworth-Parkside Community. Washington, DC: DC Promise Neighborhood Initiative. 
http://dcpni.ora/wp-content/uploads/2014/02/Final-l-Heart-Data-Booklet.pdf 

This booklet demonstrates several creative ways to present data using data 
visualization techniques, such as using less-traditional symbols as substitutes for 
standard conventions to express percentages and relative order of magnitude. 
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Appendix A: Aligning this Resource Guide with 
Existing Head Start National Center Program 
Management and Fi scal Operati ons ( PMFO) 
Technical Assistance Materials 

Concurrent with the development of this resource guide, PMFO has been developing 
several resources to better support data use in Head Start programs. We highlight 
two of those resources here and suggest ways of supplementing the PMFO 
materials with this resource guide. 

Using Data Effectively: The Four Data Activities Workshop 

This workshop, developed by PMFO, is organized around the four data activities of 
prepare, collect, aggregate and analyze, and use and share. The workshop has two 
intended outcomes: to enable participants to (1) "tell why data is important," and 
(2) "describe the four data activities, and use them with their own program's data." 

Following are some suggestions about how to use the "Moving Beyond 
Compliance" resource guide developed by the Urban Institute to supplement the 
PMFO workshop resources: 

1. Before the workshop, send Part I of the resource guide to intended participants. 
Ask them to read "The Changing Compliance and Performance Landscape" and 
"Moving from Compliance to Learning in Head Start." These two sections will help 
situate the importance of the workshop activities in the context of changes in Head 
Start. The rest of Part I may also be helpful for reading in advance of the workshop, 
but trainers should consider the interest level of their audience. 

2. At the end of the first day of the workshop, hand out Part 1 1 of the resource 
guide. Part II focuses on making the culture shift inside organizations. It will help 
participants think about where their organizations are along the learning continuum 
and will help them see more possibilities in the discussions on day two for how they 
might implement data changes in their organizations. 

3. During Day 2 "Aggregate and Analyze" discussions, consider sharing 
"Disaggregating Your Data," particularly Table C, from Part IV of the resource guide 
to supplement the existing materials. 

4. When you reach Trainer's Note #56: From Individual to Program, share the 
handout "Using Data for Internal Program Improvement" from Part III of the 
resource guide. This handout will help participants think about the numerous ways 
that program level data could be useful to them. 
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5. When you reach Trainer's Note #110 about the Stephen Covey video clip, hand 
out Appendix B: Continuous Quality Improvement Conceptual Framework from the 
resource guide. Use the framework to facilitate the discussion of the video. Point 
out how the literature from numerous fields all point toward the importance of a 
Culture of Collaborative Inquiry (the team), and leadership and clear 
communication (leadership slab). You may also want to hand out "From Fighting 
Fires to Innovation: An Analogy for Learning" from the resource guide. 

6. At the end of the workshop, remind participants that they can find many 
supplemental resources in the resource guide including more examples on how to 
use data for program improvement and how to develop a cycle of continuous 
improvement; see especially Part III: Using Data for Internal Program 

I mprovement. 

7. Your participants are bound to ask you questions about how they can include 
multiple participants into the four data activities, and how they can build data into 
their day. You can refer them to Part V: Engaging Staff and Stakeholders for some 
ideas. 

Planning in Head Start: The Program Planning Papers — 

Developed by the Head Start National Centers, the Program Planning Papers cover 
five topics: (1) Understanding Goals, Objectives, Outcomes, Progress, and Action 
Plans; (2) Plans in Head Start; (3) Program Goals and School Readiness Goals - 
Understanding the Relationship; (4) Goals, Objectives, Outcomes, Progress, and 
Action Plans - Some Examples; and (5) Program Planning and Parent, Family, and 
Community Engagement. 

The resource guide provides materials that may be used to supplement the 
Program Planning Papers. The resource guide authors suggest the following 
alignment and supplementary usage. 

Topic #1: Understanding Goals, Objectives, Outcomes, Progress, and Action Plans 
Read Part I : Background of the resource guide in conjunction with Topic #1 
of the Planning Papers. This part of the resource guide will help readers 
understand why Head Start staff should care about planning and data use in 
Head Start and how the federal and Head Start philosophies have changed 
overall. 

Topic #2: Plans in Head Start 

Read Part III: Using Data for Internal Program Improvement of the resource 
guide. The resource guide provides a continuous improvement cycle that 


39 PMFO, Foundations for Excellence. 
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helps grantees to see how they might create systems within their programs 
to facilitate the planning cycle and integrate the information collected into 
regular program use. The resource guide outlines a system and provides a 
few examples. 

Topic #3: Program Goals and School Readiness Goals - Understanding the 

Relationship 

- The resource guide does not specifically deal with this topic, but it does 
provide examples of data disaggregation which may help in thinking about 
goals at different levels. See especially Part IV, Table C: "Possible Client and 
Service Characteristics to Use in Disaggregating Data." 

Topic #4: Goals, Objectives, Outcomes, Progress, and Action Plans - Some 

Examples 

The resource guide provides a few examples for planning and data use. See 
Part III. 

Topic #5: Program Planning and Parent, Family, and Community Engagement 
Part 1 1 and Part V of the resource guide provide important considerations. 
Part II: The Culture Shift to Continuous Improvement describes 
organizational learning cultures and helps grantees to reflect on the extent 
that they include many individuals in the learning process. Part V: Engaging 
Staff and Stakeholders provides some specific examples of how to create 
internal processes that are inclusive of many voices. 


65 



Appendi x B - Conti nuous Quali ty I improvement 

Conceptual Framework 


The LEADS study is grounded in a multidisciplinary review of the literature 40 on the 
processes, facilitators, and impediments to data use for continuous quality 
improvement. That review reflects more than 50 seminal and current works that 
originate in empirical and professional sources in the fields of educational leadership 
and management, health care management, nonprofit leadership and management, 
public management, and organizational learning and development. Studies in the 
literature review were typically designed to identify characteristics of organizations 
or programs that were successful in implementing data use for quality 
improvement, but they do not explore the relative importance of each characteristic 
or the differing effects of combinations of characteristics. 

The study of organizational data use does not have a single language across 
fields. The terms performance management, continuous quality improvement, and 
data- informed decision-making are all descriptors of the internal organizational 
processes, functions, and elements for collecting, examining, and using data to 
improve program performance. This brief uses the term continuous quality 
improvement to reduce confusion and to emphasize the focus on data use for 
improving quality. 

Eight organizational elements emerged from the literature as supporting data 
use for continuous quality improvement. These eight elements were integrated to 
construct a conceptual framework (see figure E). Specifically, the conceptual 
framework depicts the key elements posited to facilitate or impede the process of 
data use for continuous quality improvement in the form of a building. The 
framework is cautious in its representation of the relationships of elements to each 
other. It makes minimal use of directional arrows because the literature from other 
fields is primarily descriptive, cataloging elements but not relationships between 
elements. 

Strong leadership is one of the two most common themes cited in the 
literature as influencing data use. Program leadership ensures the organization has 
the required resources, analytic capacity, and professional development to use 
data. Specifically, the literature identifies certain leadership approaches (e.g., 
leadership that is distributed across staff) as important to building organizational 
features that are facilitators of data use (e.g., culture of collaborative inquiry). For 
this reason, leadership is shown as the foundation slab of the building. 


40 Derrick-Mills et al. (forthcoming). 
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The pillars of the building represent the important facilitative supports 
leaders can put into place: commitment of resources, analytic capacity, and 
professional development. The literature suggests these factors are associated with 
the effective use of data, and the absence of any of these factors is likely to reduce 
an organization's ability to continuously and successfully use data for quality 
improvement. The pillars and foundation support a culture conducive to 
collaborative inquiry where staff learn together in an environment that fosters joint 
problem-solving, creativity, and innovation. The roof of the building represents the 
continuous cycle of data use, or data-informed decision-making. 
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Figure E. Continuous Quality Improvement Conceptual Framework 
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Appendix C— Head Start Frameworks for 
Continuous Program Improvement 


The Head Start national centers have developed many cycles representing the 
continuous data use process. We provide some of them here. Each cycle looks 
slightly different and each uses different words to describe its process, but these 
cycles and the one we reference in the resource guide are designed with the same 
general purpose in mind: 

(1) Emphasizing the importance of continuous data use 

(2) Showing that data use involves a systematic process 

The cycle in the resource guide is a data-use cycle that can be used with any 
type of data in any type of setting. The specific cycles displayed here are designed 
to help you think specifically about particular kinds of data required by Head Start. 
It can be helpful to think of these cycles in both generic and specific terms. 

Data in Head Start and Early Head Start: Creating a Culture that 
Embraces Data: Program Planning Cycle 



Goals/Communicate I 
to Stakeholders 



>> 


if »i i \ < fiiTFi i ci i» i 


with Course 
Corrections i 






A 




L, 


Implement Plan of 
Action 


Source: National Center on Program Management and Fiscal Operations, n.d. Data in Head 
Start and Early Head Start: Creating a Culture that Embraces Data. Washington, DC: US 
Department of Health and Human Services, Administration for Children and Families, Office 
of Head Start, http://eclkc.ohs.acf.hhs.gov/hslc/tta-svstem/operations/center/data/data- 
auide.pdf . 
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Program Planning: Spotlight on Fiscal Data 

Program Planning Cycle 



Source: National Center on Program Management and Fiscal Operations, n.d. Program 
Planning in Head Start: The Program Planning Cycle. Washington, DC: US Department of 
Health and Human Services, Administration for Children and Families, Office of Head Start. 
https://eclkc.ohs.acf.hhs.aov/hslc/tta-svstem/operations/docs/plannina-cvcle.pdf . 
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PMFO, Planning in Head Start, 2014 


Integrating Data into Program Planning Systems/ 



Community Assessment 

Analyze data 
Share conclusions 
Communicate results to internal 
and external audiences 


Evaluate Progress Through 
Self-Assessment 

Assess annual progress in achieving goals 
and objectives 

Assess effectiveness of systems and 
services 

Examine trends and patterns 
Communicate results to internal and 
external audiences 


m 

°*m Cl* 


* THE NATIONAL CENTER ON 

Program Management 
and Fiscal Operations 


Decide on Goals/ Communicate 
with Stakeholders 


Ensure goals reflect conclusions 
from key data sources, .e.g., 
community assessment, child 
records 


Continually Respond with Mid- 
Course Corrections 


Use Record-keeping Reporting 
System to collect data 
Check integrity of data 
Communicate data findings and 
next steps to internal audiences 



Evaluate Progress Through 
Ongoing Monitoring 


Aggregate data and review for overall 
trends monthly 
Analyze data monthly (e.g., by 
center, option, position, etc.) 

Draw conclusions 
Communicate findings 


o* 

JV. THE national center on 
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V UV £ and Fiscal Operations 
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Develop Plan of Action 
and Budget that Reflect Goals 

Determine evidence 
Determine data collection 
methodologies 

Adjust Record-keeping & Reporting 
Systems 


H 


Implement Plan of Action 


Use Record-keeping & Reporting 
System to collect data 
Check integrity of data, e.g., 
supervisors do spot checks of 
reports 

Discuss reports at regularly 
scheduled intervals 


Source: National Center on Program Management and Fiscal Operations, n.d. Program 
Planning in Head Start: The Program Planning Cycle. Washington, DC: US Department of 
Health and Human Services, Administration for Children and Families, Office of Head Start. 
https://eclkc.ohs.acf.hhs.aov/hslc/tta-svstem/operations/docs/plannina-cvcle.pdf . 
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Introduction to Data Analysis Handbook, Migrant and Seasonal 
Head Start Technical Assistance Center, Academy for Educational 

Development 


Data Analysis: In Cyclical Terms * Feedback Loop 


Writing: 

How Do We 
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of the Data? 


Reporting 
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Interpretation: 
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Findings: 
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Analysis: 
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Collection: 
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Answer The Question? 


PURPOSE(S) 

What You Do ft The Reasons Why 


Source: Migrant and Seasonal Head Start Assistance Center. 2006. Introduction to Data 
Analysis Handbook. Washington, DC: Academy for Educational Development. 
http://ece.aed.org/publications/mshs/dataanalvsis/WebDataAnalvsis.pdf 
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How Child Outcomes Assessment Supports Continuous Program 
Improvement: Head Start Child Outcomes 

Continuous Improvement Cycle 
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Comnnunily Assessment 


Source: National Center on Quality Teaching and Learning. "How Child Outcomes 
Assessment Supports Continuous Program Improvement." Office of Head Start, last 
modified October 7, 2014, http://eclkc.ohs.acf.hhs.gov/hslc/tta- 
svstem/teachinq/eecd/Curriculum/Planninq/edudev art 00091 080905.html . 
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